STFM Annual Spring Conference, Denver Evaluating impact of a multicultural interclerkship on students’ knowledge, skills and awareness Warren J. Ferguson, MD Michael Godkin, PhD Heather Lyn Haley, PhD Mary Lindholm, MD Mary Philbin, EdD STFM Annual Spring Conference, Denver April, 29, 2009 Email: fergusow@umassmemorial.org

Objectives By the end of this session, participants will: Be aware of the lack of evaluation of cultural curriculums Understand the curriculum we developed to teach cultural competency and the tools used to evaluate its efficacy Help develop next steps for culture curriculum and its evaluation, sharing experiences at their own institutions with attention to evaluation methods being utilized

Background: Health Policy Considerations 2006: California requires all CME to include a thread on cultural and linguistic considerations 2008: New Jersey requires all untrained physicians to document 6 hours of cultural competence CME BC/BS of MA and FL have made cultural competence CME a condition for P4P

Background on Student Education 2000: LCME sets standards for cultural competence curriculum (ED 21 & ED 22) 2005: AAMC develops TACCT Domains Specific Components http://www.aamc.org/meded/tacct/culturalcomped.pdf

What about evaluation? Conceptual framework for evaluation KSA Two systematic literature reviews

Most studies show positive effect Source: Beach MC et al. Med Care (2005) 43: 356-73

Quality of evaluation methods lacking Systematic review with 64/288 reviewed met inclusion criteria Assessed five domains of quality Description of subjects and settings (21/64) Complete description of intervention (21/64) Use of comparison groups (8/64) Objective outcome assessment (27/64) Report of analytic approach (15/64) Source: Price EG et al. Acad Med 2005 80:578-86

Your experiences with cultural curricula and evaluation

UMMS Multicultural Interclerkship Interclerkship day = one day workshop One of nine required topics for MS 3 Domestic Abuse Chronic Pain; Oral Health Health Policy & Medical Practice Geriatrics End of Life Medical Error and Patient Safety Disabilities Multiculturalism Interclerkship started in 2001

Course Objectives Complete an ethnographic interview. Develop cultural sensitivity to the hardships faced by immigrants and refugees during the migration process. Develop knowledge about cultural variations in the health beliefs, expressions and practices of patients, and provider care and outcomes. Develop knowledge about intervention and management techniques for working with other cultures. Develop knowledge about the ways racial issues affect health care outcomes and access of African-Americans.

Curriculum Introduction Health Disparities and Population Diversity Plenary Ethnographic Interviewing & Demonstration Class Half A (2 workshops) GLBT care or Sentinel Cross Cultural Problems Health Disparities in ED or Culture and Mental Health Class Half B-Ethnographic Interviews Small groups of 8 students Faculty Facilitator Four ethnographic interviews Lunch Class Half A-Ethnographic Interviews Class Half B-Workshops

Elements of Evaluation 2001-2009 Course Rating by Participants 2001-2009 Pre-Post KSA Survey of Participants 2005-2009 End of 4th Year KSA Survey 2006-2007 Pilot OSCE

Pre-Post Survey Demographics Age, Gender, Race/Ethnicity Immigration status Optional Enrichment Pathway on Serving Multicultural and Underserved Populations Optional Enrichment International Learning Career Preference

Instrument Elements Instrument developed based on course objectives Three domains Attitudes: 5 questions Skills: 10 questions Knowledge: 8 questions Four point Likert scale(1-4) Strongly disagree-Disagree-Agree-Strongly Agree Data analyzed using Paired Sample t Test in SPSS

Results from 2007 (n=102) Pre Post Post-Pre t Sig. (2-tailed 2.79 3.15 Domain Pre Post Post-Pre t Sig. (2-tailed Attitude 2.79 3.15 0.37 9.59 .000 Skill 2.91 3.12 0.21 7.16 Knowledge 2.83 3.07 0.24 7.48

Do changes in selected instrument scores persist? Negotiated with Dean’s office to include selected questions from instrument on Graduation Questionnaire Initiated with Class of 2006 Interclerkship Course 2005 Analyzed data from Class of 2006 using Repeated Measures Test in SPSS

Results (N=56) Domain Pre Post* Grad** Attitude 2.83 3.13 3.25 Skill 2.77 3.04 Knowledge 2.71 3.06 3.17 *All results significant **All results significant compared to pre

Behavioral Skills Assessment Hypothesis: Given significant changes in pre-post instrument scores, performance on an OSCE designed to measure performance on ethnographic interviewing will improve following the interclerkship Pilot test of a one station OSCE using a convenience sample of MS3 in FM clerkship pre- and post-interclerkship Ungraded, formative learning task Applied and was exempt from IRB review

Design July 05 October 05 Jan 06 Apr 06 June 06 Pilot Test OSCE Block 2 2006 Interclerkship Design OSCE Recruit & train SPs Design evaluation instrument Pre-Interclerkship OSCE Blocks 3A and 3B Post-Interclerkship OSCE Blocks 4A and 4B

OSCE Case Middle-aged Liberian immigrant Chief Complaint: Lack of energy and difficulty sleeping, getting worse No physical symptoms suggestive of medical problem Rich social history of immigration and history of significant loss and trauma

Pilot OSCE tasks Recruit standardized patients Have SPs evaluate the case as written Train standardized patients Schedule sessions in two FM blocks Arrange for videotaping Develop rating scale Three faculty reviewed 5 videos along with communication skills trainer Edited the rating scale Test for inter-rater reliability Raters blinded to pre/post interclerkship students

Inter-rater reliability testing Subscale Sample/raters ICC CI p Socio/cultural content Basic elements 5/4 .953 .763-.997 .000 Advanced elements ..965 .825-.998 Interview Process .883 .407-.992 .005 .697 .539-.979 .072

Pilot OSCE results Subscale Untrained n=14 Trained n=8 F p Socio/cultural content Basic elements 1.51 (.19) 1.55 (.28) .13 >0.25 Advanced elements .89 (.32) .70 (.44) 1.23 Interview Process 1.71 (.19) 1.67 (.19) .21 1.57 (.40) 1.55 (.34) .01 Analysis of variance between trained and untrained students (on a scale where 0=not addressed – 2=addressed fully)

Limitations of Study Small sample Lack of consistency between SPs Convenience sample No randomization

Next Steps Multi-year data analysis Pre/post/post Regression analysis of demographic factors and performance on KSA instrument

Conclusions One day workshop has been a successful venue for cultural competence training Small group work is most effective Pre-post improvement on self-assessed attitudes, skills and knowledge Change persists in all three domains at end of fourth year Pilot OSCE demonstrated no significant behavior change