1. DAVID V POWER MD MPH THERESE M ZINK MD MPH KEN P OLSON MD ILENE B HARRIS PHD A qualitative review of students’ clinical performance STFM Annual Meeting 2010

2. Question How do we measure whether there are differences in ‘comfort managing outpatient clinic encounters’ between two groups of students who have followed different curricula? o ‘Comfort in clinic encounters’ – an evasive variable

3. Hypothesis Students who have spent nine months working as apprentices in rural primary care clinics (RPAP) appear: ‘better at handling themselves with patients’ ‘more comfortable in clinic’ but... There are minimal quantitative differences between the groups on academic measures (MCAT, USMLE etc.) Quantitative scoring is equivalent on an OSCE

4. RPAP Founded in 1971 with Minnesota legislature funding to increase the numbers of rural primary care doctors 3rd year medical students (35-40/year) in a rural community with a primary care preceptor (90%+ FM) for 9-months A longitudinal, continuity hands-on immersion experience Generally interested in primary care and / or rural RPAP students desirable first year residents

5. Non-RPAP 8 weeks outpatient medicine All students (15% FM) Much fewer clinic patient contact hours  (240 v 800 estimate) Many fewer procedures End of clerkship OSCE: comprehensive, challenging

6. Qualitative Method (IH) 1. Review performances 2. Identify general themes for comparison 3. Identify sub-themes within main themes 4. Reach consensus on main themes 5. Review all performances for the presence or absence of these themes 6. Compare frequency of observations between the two groups Reference: Fraenkel JR, Wallen NE. How to design and evaluate research in eduation. McGraw-Hill, 2008

7. Method – Sample Video-taped OSCE encounters for RPAP students (40) and 2 sets of non-RPAP clerkship students (65) Students consented to participation (all but a few) Selected four ‘representative’ stations Identified quantitatively high AND low scoring students on each station from both groups

8. Method- Faculty Raters 3 MD faculty: 1 RPAP faculty (TZ), 1 clerkship director (DP) and 1 ‘independent’ community preceptor (KO) 1 ‘blinded’ (KO), 1 ‘partly blinded’ (DP), 1 ‘not blinded’ (TZ) All with some expertise in OSCE evaluation and in ‘doctor-patient communication’ theory

9. OSCE station videotape reviews ‘Chronic cough’ (2CS format):  Patient interview  SOAP note & presentation to faculty Dysuria in female adolescent Health care maintenance in middle age male Medication reconciliation

10. Each faculty reviewed tapes independently  Identified themes  then sub-themes Small group process  Discussed each others’ observations for each scenario  Agreed upon common general themes and sub-themes Guided by a qualitative methods expert providing process advice (IH)

11. Reviewed remainder of tapes and initial ones again Applied our themes and sub-themes to analyze each taped encounter Documented presence or absence of themes / sub- themes in each encounter in a color-coded grid

12. Results - Main themes 1. Rapport Building  Introduced self  Used patient’s name  Deliberately made eye contact  Spent time talking before taking notes  Exhibited noticeably helpful non-verbal behaviors  Spent time discussing non-medical content  Reacted appropriately to patient non-medical comments  Closed the encounter as one ends a conversation

13. Results 2. Content Knowledge  Used an algorithm to guide questioning  Answered any patient questions correctly and confidently  Used medication names correctly  Knowledgeable of medication side-effects  Knowledge of testing frequency and desired goals  Appropriate development of a differential diagnosis  Effectively communicated clinical priorities for care

14. Results 3. ‘Routinisation’ of Process  Knocked on door  Comprehensive introduction statement  Relaxed, flowing delivery of information Impression of a ‘scripted’ delivery Absence of uncomfortable pauses Absence of ‘searching for what to say next’  Not bothered by answering patient questions Flexible to patient prompts Comfortable with seating arrangement  Impression of having done this many times

15. RPAP students engaged in more rapport building  RPAP students (both high and low scorers) Engaged in more informal ‘chat’ at beginning of and during the interview More consistently identified self on entering room Used patient’s name more consistently

16. RPAP students (high and low scorers) appeared better rehearsed (routinised) at explaining preventive care recommendations  Colonoscopy  Dilemma relating to PSA testing  Value of preventive care  ‘like an oil change’  One low scorer was ALL relational, limited content In comparison, low scoring TC student’s explanations  Were less flowing  Were more hesitant  Included pauses

17. TC students (high and low scoring) had more complete mastery of content of an adolescent interview  More complete teen risk assessment (using HEADDSSS)  Although one RPAP student did cover many essential areas without apparently using a script  Volunteered reassurance about confidentiality

18. Results RPAP students were more consistently good performers across board; TC students demonstrated higher variability (more highs, more lows) One RPAP student demonstrated ‘beautiful’ interpersonal skills and counseling (high on rapport and ‘routinisation’), but had content deficits with osteoporosis medications and diabetes goals identified by skilled SP, not necessarily by an ‘average’ patient

19. Investigator Biases 2 Faculty affiliated with programs Pre-existing impressions of RPAP – one positive, two mixed As teaching physicians, we know we transferred our preferences for desirable student clinic behaviors

20. Conclusions In our sample, we demonstrated:  Increased frequency Rapport building behaviors in RPAP  Increased frequency markers of ‘Routinisation’ in RPAP  Increased frequency Content markers especially in one specific content area (adolescent interview) non-RPAP  Overall more consistently positive performances among RPAP students

21. Implications - Process Qualitative assessment of OSCEs reveals competencies not captured in quantitative scoring Seasoned, well-trained SPs no longer appear to evaluate the same way as untrained patients might Faculty MDs impressions did not consistently correlate with quantitative scores

22. Implications - Content High volume exposure to multiple clinic patient encounters (as in RPAP) facilitates increases all students comfort with managing patients in outpatient setting Performance in less commonly encountered clinical situations (adolescent interview) are improved by education in this topic and providing students a script to use in practice