1. STFM Annual Spring Conference, Denver
Evaluating impact of a multicultural interclerkship on students’ knowledge, skills and awareness
Warren J. Ferguson, MD
Michael Godkin, PhD
Heather Lyn Haley, PhD
Mary Lindholm, MD
Mary Philbin, EdD
STFM Annual Spring Conference, Denver
April, 29, 2009

2. Objectives By the end of this session, participants will:
Be aware of the lack of evaluation of cultural curriculums
Understand the curriculum we developed to teach cultural competency and the tools used to evaluate its efficacy
Help develop next steps for culture curriculum and its evaluation, sharing experiences at their own institutions with attention to evaluation methods being utilized

3. Background: Health Policy Considerations
2006: California requires all CME to include a thread on cultural and linguistic considerations
2008: New Jersey requires all untrained physicians to document 6 hours of cultural competence CME
BC/BS of MA and FL have made cultural competence CME a condition for P4P

4. Background on Student Education
2000: LCME sets standards for cultural competence curriculum (ED 21 & ED 22)
2005: AAMC develops TACCT
Domains
Specific Components

5. What about evaluation? Conceptual framework for evaluation
KSA
Two systematic literature reviews

6. Most studies show positive effect
Source: Beach MC et al. Med Care (2005) 43:

7. Quality of evaluation methods lacking
Systematic review with 64/288 reviewed met inclusion criteria
Assessed five domains of quality
Description of subjects and settings (21/64)
Complete description of intervention (21/64)
Use of comparison groups (8/64)
Objective outcome assessment (27/64)
Report of analytic approach (15/64)
Source: Price EG et al. Acad Med :578-86

8. Your experiences with cultural curricula and evaluation

9. UMMS Multicultural Interclerkship
Interclerkship day = one day workshop
One of nine required topics for MS 3
Domestic Abuse
Chronic Pain; Oral Health
Health Policy & Medical Practice
Geriatrics
End of Life
Medical Error and Patient Safety
Disabilities
Multiculturalism Interclerkship started in 2001

10. Course Objectives Complete an ethnographic interview.
Develop cultural sensitivity to the hardships faced by immigrants and refugees during the migration process.
Develop knowledge about cultural variations in the health beliefs, expressions and practices of patients, and provider care and outcomes.
Develop knowledge about intervention and management techniques for working with other cultures.
Develop knowledge about the ways racial issues affect health care outcomes and access of African-Americans.

11. Curriculum Introduction Health Disparities and Population Diversity
Plenary
Ethnographic Interviewing & Demonstration
Class Half A (2 workshops)
GLBT care or Sentinel Cross Cultural Problems
Health Disparities in ED or Culture and Mental Health
Class Half B-Ethnographic Interviews
Small groups of 8 students
Faculty Facilitator
Four ethnographic interviews
Lunch
Class Half A-Ethnographic Interviews
Class Half B-Workshops

12. Elements of Evaluation
Course Rating by Participants
Pre-Post KSA Survey of Participants
End of 4th Year KSA Survey
Pilot OSCE

13. Pre-Post Survey Demographics Age, Gender, Race/Ethnicity
Immigration status
Optional Enrichment Pathway on Serving Multicultural and Underserved Populations
Optional Enrichment International Learning
Career Preference

14. Instrument Elements Instrument developed based on course objectives
Three domains
Attitudes: 5 questions
Skills: 10 questions
Knowledge: 8 questions
Four point Likert scale(1-4)
Strongly disagree-Disagree-Agree-Strongly Agree
Data analyzed using Paired Sample t Test in SPSS

15. Results from 2007 (n=102) Pre Post Post-Pre t Sig. (2-tailed 2.79 3.15
Domain
Pre
Post
Post-Pre t
Sig. (2-tailed
Attitude
2.79
3.15
0.37
9.59
.000
Skill
2.91
3.12
0.21
7.16
Knowledge
2.83
3.07
0.24
7.48

16. Do changes in selected instrument scores persist?
Negotiated with Dean’s office to include selected questions from instrument on Graduation Questionnaire
Initiated with Class of 2006
Interclerkship Course 2005
Analyzed data from Class of 2006 using Repeated Measures Test in SPSS

17. Results (N=56) Domain Pre Post* Grad** Attitude 2.83 3.13 3.25 Skill
2.77
3.04
Knowledge
2.71
3.06
3.17
*All results significant
**All results significant compared to pre

18. Behavioral Skills Assessment
Hypothesis: Given significant changes in pre-post instrument scores, performance on an OSCE designed to measure performance on ethnographic interviewing will improve following the interclerkship
Pilot test of a one station OSCE using a convenience sample of MS3 in FM clerkship pre- and post-interclerkship
Ungraded, formative learning task
Applied and was exempt from IRB review

19. Design July 05 October 05 Jan 06 Apr 06 June 06 Pilot Test OSCE
Block 2
2006 Interclerkship
Design OSCE
Recruit & train SPs
Design evaluation instrument
Pre-Interclerkship OSCE
Blocks 3A and 3B
Post-Interclerkship OSCE
Blocks 4A and 4B

20. OSCE Case Middle-aged Liberian immigrant
Chief Complaint: Lack of energy and difficulty sleeping, getting worse
No physical symptoms suggestive of medical problem
Rich social history of immigration and history of significant loss and trauma

21. Pilot OSCE tasks Recruit standardized patients
Have SPs evaluate the case as written
Train standardized patients
Schedule sessions in two FM blocks
Arrange for videotaping
Develop rating scale
Three faculty reviewed 5 videos along with communication skills trainer
Edited the rating scale
Test for inter-rater reliability
Raters blinded to pre/post interclerkship students

22. Inter-rater reliability testing
Subscale
Sample/raters
ICC CI p
Socio/cultural content
Basic elements
5/4
.953
.000
Advanced elements
..965
Interview Process
.883
.005
.697
.072

23. Pilot OSCE results Subscale Untrained n=14 Trained n=8 F p
Socio/cultural content
Basic elements
1.51 (.19)
1.55 (.28)
.13
>0.25
Advanced elements
.89 (.32)
.70 (.44)
1.23
Interview Process
1.71 (.19)
1.67 (.19)
.21
1.57 (.40)
1.55 (.34)
.01
Analysis of variance between trained and untrained students
(on a scale where 0=not addressed – 2=addressed fully)

24. Limitations of Study Small sample Lack of consistency between SPs
Convenience sample
No randomization

25. Next Steps Multi-year data analysis Pre/post/post
Regression analysis of demographic factors and performance on KSA instrument

26. Conclusions
One day workshop has been a successful venue for cultural competence training
Small group work is most effective
Pre-post improvement on self-assessed attitudes, skills and knowledge
Change persists in all three domains at end of fourth year
Pilot OSCE demonstrated no significant behavior change