A Practical Approach to Evaluation in the Ambulatory Setting in the Era of the New ACGME General Competencies Eric S. Holmboe Stephen Huot Yale University School of Medicine Yale Primary Care Residency Program

ACGME Core Competencies Medical knowledge Patient care Practice-based learning and improvement Interpersonal and communication skills Professionalism Systems-based practice

Workshop Objectives Understand the importance of the outpatient setting for assessment of clinical skills Appreciate importance of directly observing residents interacting with patients Discuss practical strategies for focused direct observation

Workshop Elements Mini-Lectures: Direct observation exercises Basic Premises Ambulatory clinical skills Faculty rating accuracy Direct observation exercises Performance dimension exercise Videotape evaluation exercises

Basic Premises Accurate resident evaluation – important Decision-making – “summative” Feedback – “formative” Professional obligation Resident observation Traditional and vital

Ambulatory Clinical Skills History taking Focused physical examinations Counseling and education Reflective practice

Importance of Sound Clinical Skills Physician behaviors and communication Accuracy / completeness of data gathering Patient satisfaction and compliance Clinical outcomes Legal implications Contribution of History & PE to decision-making 80 to 90% diagnoses made by H & P Cost-effective use of health care resources

Clinical Skills Stillman (1990) Sachdeva (1995) Wide variability in MS4 clinical skills Sachdeva (1995) Wide variability in intern skills Wray (1983) / Johnson (1986) High frequency of errors Mangione (1997) Deficient cardiac auscultatory skills

Clinical Skills Suchman (1997) Ramsey (1998) Braddock (1999) Poor communication / humanistic skills Ramsey (1998) Incomplete history-taking / preventive health screening Braddock (1999) Of > 1000 patient visits, less than 15% fulfilled core elements of informed decision making

Resident Clinical Skills: Themes Deficiencies exist across continuum Specific skills more “error-prone” Not detected by other evaluation methods Basic clinic skills don’t correlate with other competence dimensions Residents aware of importance and under-emphasis Without detection cannot be corrected

ACGME and Direct Observation Direct Observation crucial to evaluate: Patient care History taking, Pexam, counseling Interpersonal and communication skills Patient/peer/colleague interactions Professionalism

Faculty Observation / Rating Skills Thompson (1990)/Haber (1994) Significant “halo effect” with ratings Ratings based mostly on perceived knowledge and personality Kalet (1992) Poor reliability – interpersonal skills Poor validity and predictive value Rater training ineffective

Faculty Observation / Rating Skills Herbers (1989) / Noel (1992) Structured > open-ended form Brief training video not effective Increased accuracy  discriminative ability Kroboth (1992) Poor inter-rater reliability Rater training ineffective

Faculty as Raters – Key Issues Faculty do not observe actual performance Faculty ratings lack: Reliability Accuracy Content specificity Under SCO – that focus on discrete skills or behaviors

Faculty as Raters - Solutions Step 1: Getting faculty to observe Required by the ACGME Focused observations are logistically possible 5 to 10 minute observations are valuable Build into existing clinic schedule Build on faculty “epiphany” The “You will not believe what I saw today” experience

Mini - CEX Tool “Structured” approach to direct observation Direct assessment of actual patient care Incorporation of CEX into daily activities High satisfaction among housestaff

Logistics: GIMC One mini-CEX per intern per day per week One attending observes portion of first visit of the day Interview, physical exam, counseling Minimizes disruption of resident clinic Perform over course of academic year Easy to obtain 6-8 Mini-CEX’s per year per intern

Faculty as Raters - Solutions Step 2: Improving reliability Multiple brief observations Perform over time: outpatient setting allows for longitudinal observation Involve multiple faculty MiniCEX: sufficient reliability for pass/fail determinations after just 4 observations

Direct Observation:Yale PGY-2 Resident Ward Onc ER ID Amb GI ICU Card HX X PE EC

Videotape Watch the following videotape and then complete a Mini-CEX evaluation on the clinical skills of this resident

Faculty as Raters - Solutions Step 3: Improve accuracy and validity Most difficult step Improved with structured rating forms Can be improved with rater training, but Brief training interventions do not work

Can You Train Faculty? Performance Appraisal Literature: Can reduce rating errors Can improve discriminative ability Can improve accuracy

Summary of Rater Training Performance Dimension Training Frame of Reference Training Behavioral Observation Training

Performance Dimension Training Involves familiarizing faculty with the specific dimensions of competence Should involve discussion of the “qualifications” required for each dimension Use the ACGME competencies and the ABIM portfolio to “calibrate” faculty

Frame of Reference Training Goal is to improve “judgment” and accuracy Steps in FOR training: 1. Raters given descriptions of each dimension - discuss “qualifications” needed for each dimension (PDT) 2. Review of clinical vignettes describing critical incidents of performance: unsatisfactory to average to superior

Frame of Reference Training 3. Raters used vignettes to then provide ratings on a behaviorally anchored rating scale (BARS) - think ABIM eval form 4. Session trainer provides feedback on what “true” ratings should be along with rationale 5. Discussion ensues about discrepancies between trainers ratings and the participants’ ratings

Frame of Reference Training Most difficult aspect of FOR: Setting the actual performance standards Reaching agreement and consensus among teaching faculty

Behavioral Observation Training Two main strategies: 1. Increase the amount of “sampling” - More observations lead to more accurate evaluations. 2. Use of observational “aides” - Behavioral diary to record observed performance.

Structuring the Observation Prepare for the observation Minimize intrusiveness – correct positioning Minimize interference with the resident-patient interaction Avoid distractions Possible solution Allow for habituation by consistent observation

Direct Observation: Challenges Like all skills, requires training and practice Faculty “calibration” important Agreeing on “metrics” of performance Faculty comfort with own skills Faculty training How, when, who, what, where

Observation Summary Sample “parts” of the visit: History-taking Physical examination Counseling Perform longitudinally No need to do it all at once Agree on performance metrics with ambulatory faculty