Monitoring & Evaluation HKMC Accreditation Site Visit 2008 Chinese University of Hong Kong HKMC Accreditation Site Visit 2008 Chinese University of Hong Kong

Monitoring & Evaluation Committee Members Dean Associate Dean (Education) Assistant Deans (Education: pre-clinical & clinical) Year Coordinators Panel & Module Coordinators Director of Office of Education Terms of Reference determine impact of curriculum, instructional & assessment design changes (begun in 2001/02) on students’ learning outcomes

Monitoring & Evaluation Committee Investigations 1. end-of-panel/module student feedback re. instructional & course quality 2. student performances in summative assessments in clinical yrs of MBChB 3. significant concern surveys of students regarding instructional quality & adequacy of curriculum coverage & clinical skill development 4. interns’ self-assessment of their developed generic skills 5. interns’ performances - 11 criteria measured every 3 months during 1-yr internship - solicited written comments from preceptors & unsolicited comments from patients during each rotation - medical incident reports involving interns during each rotation

Yrs Data Sources Interns’ Supervisors First Party Sources Third Party Sources 13 yrs 3 yrs 53 units in 7 Public Hosp 26 Schools Blinded Non blinded

Historical Controls Performances of Interns educated & trained in old curriculum ( ) new curriculum (2006, 2007)

Blinding Clinical supervisors in 1 university (teaching) hospital - were mostly aware new curriculum introduced 5 yrs previously - involved 6 hospital units Most supervisors in community (teaching affiliated) hospitals - were likely unaware of change - involved 52 hospital units

Covariates Discipline –Medicine, Paediatrics, Psychiatry, Surgery, Orthopaedics, OBG Gender –male or female intern Linear & Non linear trends –any systematic changes in supervisors’ ratings over 13 years Hospital –University (teaching) & Teaching Affiliated (community) Any interaction of covariates –with New/Old Curriculum

Results

Conservative Reporting of Results 1.Test for statistical significance (null hypothesis: “there is no improvement”) 2.If null rejected, effect size was estimated & corresponding CI calculated 3.Conclusions drawn reflect the lower bound of 95% CI for effect size 4.Used Cohen’s (1988) classification of small, medium & large effects -small effect could be meaningfully important but unless systematic investigation is undertaken, the change will be undetected -medium effect is large enough that informed personnel are aware that change has occurred -large effect is sufficiently large that lay personnel are also aware that change has occurred

Overview No statistically significant effect of clinical discipline or gender No significant interaction of clinical discipline or gender with change in curriculum Supervisors ratings tended to inflate over 13 years (linear trend) –adjustment for this covariate may have removed some effect due to new curriculum (but this is appropriately conservative)

Table VI-1 Comparing Cohorts from Old & New Curricula in Terms of Student Engagement

Student Engagement within Med Programme (part of CUHK’s assessment of its courses & programmes) At least a small beneficial gain for –interpersonal skills & group work –active learning –teaching for understanding –relationships of teachers & students –relationships among students Among the 17 attributes assessed, 16 were directionally improved

Generic Skills Development (part of CUHK’s assessment of its graduates 1 yr after graduation) Assessed these Generic Skills language proficiency, numerical competency, computer literacy, problem solving, work attitude, interpersonal and management skills Very little or no improvement in the graduates’ self-reported generic skills when cohorts educated under former & present curriculum were compared

Learning Outcomes: Blinded Trials within 4th & 5th Yr Modules Comparing performances on repeated measure summative assessments - very little or no improvement in Community Medicine - small loss in Family Medicine - large loss in OBG - very little or no improvement in Paediatrics - medium beneficial effect in Psychiatry - very little or no loss in Medicine - small beneficial effect in Surgery

Internationally Benchmarking Student Performances Using vignette based, problem solving items from the International Database for Enhanced Assessments and Learning (IDEAL) CUHK students in Medicine & Surgical 5 th final exams have achieved an average of 4% higher marks than students in the other international medical programmes

Significant Concerns Surveys Number of Significant Concerns among students, interns & alumni with curriculum & instruction in terms of the adequacy of Content Coverage - a very large beneficial effect (i.e., reduced concerns) Clinical Skill Development - a very large benefical effect (i.e., reduced concerns)

Interns’ Performances One of the best measures of impact of instructional, curriculum & assessment changes is the skill of the graduates in clinical practice Clinical supervisors in teaching affiliated hospitals rate interns educated under present curriculum better (a medium beneficial effect) in terms of - professional knowledge - clinical skill - organizational ability - clinical judgment and equivalently in terms of - attitude to patients & medical staff - willingness to learn & attendance at educational activities - use of medical language & communication skill - sense of responsibility

Conclusion: In terms of clinical judgment, there is a tendency for cohorts from new curriculum to be rated better by preceptors in teaching affiliated hospitals (& rated as similar to the old curriculum by preceptors in teaching hospitals) Example: Clinical Judgment

Interns’ Performances (cont’d) Clinical supervisors in teaching hospitals rate interns educated under present curriculum poorer (small to medium detrimental effect) in terms of professional knowledge & clinical skill - professional knowledge & clinical skill - organizational ability & sense of responsibility - attendance at educational activities & willingness to learn -communication skills & use of medical language and equivalently in terms of - attitude to patients and medical staff - attitude to patients and medical staff - clinical judgement

Example: Communication Skill Significant Interaction Conclusion: Significant Interaction in terms of their communication skill, there is a tendency for cohorts from the new & old curricula to be rated similarly by the preceptors in teaching affiliated hospitals (whereas preceptors in teaching hospitals rate cohorts from the new curriculum poorer than cohorts from the old curriculum)

Interns’ Performances (cont’d) In addition to these various specific criteria, clinical supervisors across all hospitals also provided written comments In terms of number & nature of the comments there was a a large effect in favor of how the performance of interns educated under the present curriculum were described references were made to letters from patients praising the interns

“Did a fantastic job at her first rotation, keen to learn & willing to take responsibility “One of the very best interns we ever had. He has sound general medical knowledge; hard working, reliable & totally committed to his work; well liked by his colleagues.” “Excellent attitude to patients; appreciation letters received from patients she cared for” Example Comments (cont’d)

Medical Incidence Reports Benefit of curriculum change was supported by changes in the frequency of Medical Incident Reports a 48% reduction per annum (a large beneficial effect) by interns from present curriculum compared to cohorts educated under former curriculum

Reflection

Despite achieved improvements, unachieved goals remain Vertical integration & doubling of small group teaching increased overall teaching load especially for clinical teachers Uneven tutoring in small group sessions Selective topics for special study modules require continual renewal Formative assessments administered on Web often lack formative feedback (thus are simply practice assessments rather than learning sessions) Web page support does not readily enable tracking teaching & related resource materials teaching across years in the vertically integrated programme Ownership of the curriculum not felt by entire faculty

Example Actions taken to Address Shortcomings Curriculum Centre formed & existing functional reorganized & yr committees now have responsibility for all teaching & assessments Model answers developed for tutorial sessions developed & discussed with tutors Training & aids provided to improve variety and quality of student assessments Excessive load in 5 th reduced (SSM eliminated, some coverage moved to 4 th yr)

Conflicting Results Differences between community & university hospitals could be due to Blinding changes not popular with some academic clinical teachers (extra teaching, less time for research) Differential perception by academic & non academic physicians concerning a perceived reduction in basic medical science knowledge in the newer cohorts

Limitations Internal validity of studies based on use of historical controls much less than designs using randomly assigned groups Only 2 cohorts have graduated from new curriculum need to continue to monitor

Conclusion In presence of conflicting results (i.e., differences in ratings of supervisors in community & university hospitals) –the value in having a variety of criteria, suitable for use in cross-validation studies, is confirmed

Conclusion The Faculty has monitored, evaluated & reflected on the findings from a wide variety of investigations related to the quality of its medical education programme –the information has informed on going changes as part of a policy for a continuous quality improvement –this modus of operandi will continue