1. A Practical Approach to Evaluation in the Ambulatory Setting in the Era of the New ACGME General Competencies
Eric S. Holmboe Stephen Huot
Yale University School of Medicine
Yale Primary Care Residency Program

2. ACGME Core Competencies
Medical knowledge
Patient care
Practice-based learning and improvement
Interpersonal and communication skills
Professionalism
Systems-based practice

3. Workshop Objectives
Understand the importance of the outpatient setting for assessment of clinical skills
Appreciate importance of directly observing residents interacting with patients
Discuss practical strategies for focused direct observation

4. Workshop Elements Mini-Lectures: Direct observation exercises
Basic Premises
Ambulatory clinical skills
Faculty rating accuracy
Direct observation exercises
Performance dimension exercise
Videotape evaluation exercises

5. Basic Premises Accurate resident evaluation – important
Decision-making – “summative”
Feedback – “formative”
Professional obligation
Resident observation
Traditional and vital

6. Ambulatory Clinical Skills
History taking
Focused physical examinations
Counseling and education
Reflective practice

7. Importance of Sound Clinical Skills
Physician behaviors and communication
Accuracy / completeness of data gathering
Patient satisfaction and compliance
Clinical outcomes
Legal implications
Contribution of History & PE to decision-making
80 to 90% diagnoses made by H & P
Cost-effective use of health care resources

8. Clinical Skills Stillman (1990) Sachdeva (1995)
Wide variability in MS4 clinical skills
Sachdeva (1995)
Wide variability in intern skills
Wray (1983) / Johnson (1986)
High frequency of errors
Mangione (1997)
Deficient cardiac auscultatory skills

9. Clinical Skills Suchman (1997) Ramsey (1998) Braddock (1999)
Poor communication / humanistic skills
Ramsey (1998)
Incomplete history-taking / preventive health screening
Braddock (1999)
Of > 1000 patient visits, less than 15% fulfilled core elements of informed decision making

10. Resident Clinical Skills: Themes
Deficiencies exist across continuum
Specific skills more “error-prone”
Not detected by other evaluation methods
Basic clinic skills don’t correlate with other competence dimensions
Residents aware of importance and under-emphasis
Without detection cannot be corrected

11. ACGME and Direct Observation
Direct Observation crucial to evaluate:
Patient care
History taking, Pexam, counseling
Interpersonal and communication skills
Patient/peer/colleague interactions
Professionalism

12. Faculty Observation / Rating Skills
Thompson (1990)/Haber (1994)
Significant “halo effect” with ratings
Ratings based mostly on perceived knowledge and personality
Kalet (1992)
Poor reliability – interpersonal skills
Poor validity and predictive value
Rater training ineffective

13. Faculty Observation / Rating Skills
Herbers (1989) / Noel (1992)
Structured > open-ended form
Brief training video not effective
Increased accuracy  discriminative ability
Kroboth (1992)
Poor inter-rater reliability
Rater training ineffective

14. Faculty as Raters – Key Issues
Faculty do not observe actual performance
Faculty ratings lack:
Reliability
Accuracy
Content specificity
Under SCO – that focus on discrete skills or behaviors

15. Faculty as Raters - Solutions
Step 1: Getting faculty to observe
Required by the ACGME
Focused observations are logistically possible
5 to 10 minute observations are valuable
Build into existing clinic schedule
Build on faculty “epiphany”
The “You will not believe what I saw today” experience

16. Mini - CEX Tool “Structured” approach to direct observation
Direct assessment of actual patient care
Incorporation of CEX into daily activities
High satisfaction among housestaff

17. Logistics: GIMC One mini-CEX per intern per day per week
One attending observes portion of first visit of the day
Interview, physical exam, counseling
Minimizes disruption of resident clinic
Perform over course of academic year
Easy to obtain 6-8 Mini-CEX’s per year per intern

18. Faculty as Raters - Solutions
Step 2: Improving reliability
Multiple brief observations
Perform over time: outpatient setting allows for longitudinal observation
Involve multiple faculty
MiniCEX: sufficient reliability for pass/fail determinations after just 4 observations

19. Direct Observation:Yale PGY-2 Resident
Ward
Onc ER ID
Amb GI
ICU
Card HX X PE EC

20. Videotape
Watch the following videotape and then complete a Mini-CEX evaluation on the clinical skills of this resident

21. Faculty as Raters - Solutions
Step 3: Improve accuracy and validity
Most difficult step
Improved with structured rating forms
Can be improved with rater training, but
Brief training interventions do not work

22. Can You Train Faculty? Performance Appraisal Literature:
Can reduce rating errors
Can improve discriminative ability
Can improve accuracy

23. Summary of Rater Training
Performance Dimension Training
Frame of Reference Training
Behavioral Observation Training

24. Performance Dimension Training
Involves familiarizing faculty with the specific dimensions of competence
Should involve discussion of the “qualifications” required for each dimension
Use the ACGME competencies and the ABIM portfolio to “calibrate” faculty

25. Frame of Reference Training
Goal is to improve “judgment” and accuracy
Steps in FOR training:
1. Raters given descriptions of each dimension - discuss “qualifications” needed for each dimension (PDT)
2. Review of clinical vignettes describing critical incidents of performance: unsatisfactory to average to superior

26. Frame of Reference Training
3. Raters used vignettes to then provide ratings on a behaviorally anchored rating scale (BARS) - think ABIM eval form
4. Session trainer provides feedback on what “true” ratings should be along with rationale
5. Discussion ensues about discrepancies between trainers ratings and the participants’ ratings

27. Frame of Reference Training
Most difficult aspect of FOR:
Setting the actual performance standards
Reaching agreement and consensus among teaching faculty

28. Behavioral Observation Training
Two main strategies:
1. Increase the amount of “sampling”
- More observations lead to more accurate evaluations.
2. Use of observational “aides”
- Behavioral diary to record observed performance.

29. Structuring the Observation
Prepare for the observation
Minimize intrusiveness – correct positioning
Minimize interference with the resident-patient interaction
Avoid distractions
Possible solution
Allow for habituation by consistent observation

30. Direct Observation: Challenges
Like all skills, requires training and practice
Faculty “calibration” important
Agreeing on “metrics” of performance
Faculty comfort with own skills
Faculty training
How, when, who, what, where

31. Observation Summary Sample “parts” of the visit:
History-taking
Physical examination
Counseling
Perform longitudinally
No need to do it all at once
Agree on performance metrics with ambulatory faculty