1. The MiniCEX and the Evaluation of Clinical Skills National Health Service Foundation Training Program

2. Patient Care Competencies Gather accurate, essential information from all sources, including medical interviews, physical examinations, medical records and diagnostic/therapeutic procedures Make informed recommendations about preventive, diagnostic and therapeutic options and interventions that are based on clinical judgment, scientific evidence, and patient preference.

3. Patient Care: Themes Clinical skills essential to patient care Cannot make “good” decisions unless you work with good and accurate information –GIGO principle Evaluation of clinical skills requires direct observation

4. Workshop Objectives Review current state of –Physician clinical skills –Faculty evaluation skills Understand the importance of direct observation by faculty for assessment of clinical skills

5. Workshop Objectives Discuss practical strategies for focused direct observation Review rater training methods –Direct Observation of Competence (DOC) training

6. Workshop Elements Mini-Lectures: –State of clinical skills –Quality of faculty ratings Direct observation exercises –Performance dimension exercise –Frame of reference training

7. Video Exercise Situation: A foundation trainee performs a medical interview in the outpatient setting. Using the MiniCEX form provided, please rate the performance of this trainee.

8. Key “Basic” Clinical Skills Medical interviewing Physical examinations Counseling/patient education Clinical judgment/reasoning Reflective practice –Self-directed learning –Professional growth and improvement –Medical errors

9. Are Clinical Skills Important? Where do clinical skills fall into the hierarchy of physician competencies and mastery in an era of advanced technology?

10. Diagnosis and Medical Interview Hampton (BMJ, 1975): –Medical interview: 82% –Physical exam: 9% –Laboratory: 9% Kirch (Medicine, 1996) –Medical interview (+PE): 70% –Imaging: 35%

11. Importance of Sound Clinical Skills Diagnostic errors –Inaccurate/ incomplete medical interview one of leading causes (Bordage) Wrong information leads to wrong decisions Patient satisfaction –Higher with better communication skills Patient self-care –Better adherence and outcomes associated with better physician communication skills

12. Clinical Reasoning: A Primer Patient/situation characteristics Prior knowledge Problem Representation* Information Gathering Context EvaluationAction Gruppen and Frohna, International Handbook on Research, 2002

13. Clinical Skills: U.K. Trainees Fox (2000) – Voluntary study of 22 PRHOs using OSCE – Only 45% with passing score on drug advice communication station – 0% passed locomotor system examination Evans (2004) –26 new PHROs –All had passed 22 station OSCE in medical school –Majority failed skill stations in blood pressure measurement and cannulation

14. Clinical Skills: U.S. Trainees Stillman (1990) –OSCE: wide variability in graduating medical student clinical skills Mangione (1997) –Deficient cardiac and pulmonary auscultatory skills –Medical students, FP and IM residents –Replicated findings in Canada and U.K.

15. Clinical Skills: Practicing MDs Ramsey (1998) –Incomplete history-taking / preventive health screening among Primary care physicians in Northwest Unites States Braddock (1999) – Study of informed decision making (IDM) and counseling – Simple analysis of presence or absence of 7 key elements –1058 outpatient visits: only 9% of visits met minimal criteria for IDM

16. Importance of Faculty: U.K. Studies Grant (Med Educ, 2003) – Inadequate coverage and frequency of supervision activities – Discordance between specialist registrars and attendings Kilminster (Med Educ, 2000) – Systematic review of supervision – Better supervision associated with improved patient safety and quality of care.

17. Importance of Faculty: U.S. Studies Inpatient Study (Lancet, 2003) – Reviewed 100 consecutive admissions to teaching service in U.S. – Faculty detected 26 PExam findings missed by house officer that impacted patient’s care Outpatient Studies – Two separate studies showed that faculty assessment disagreed with that of house officer in up to 30% of patients

18. Clinical Skills: Themes Deficiencies exist across continuum Specific skills more “error-prone” –Eg: musculoskeletal and neuro exams Not detected by other evaluation methods –Performance of basic clinic skills does not correlate with performance in other dimensions of competence

19. Clinical Skills: Themes House officers: –Aware of importance –Recognize under-emphasis Without detection deficiencies in clinical skills cannot be corrected

20. Miller’s Pyramid KNOWS KNOWS HOW SHOWS HOW DOES MCQ EXAM Extended matching / CRQ OSCE Portfolios Faculty Observation

21. Faculty Observation / Rating Skills Patient care settings –Ratings based mostly on perceived knowledge and personality –Little evidence of direct observation –Significant “Halo” effect Gray, Thompson, Haber, Grant, etc.

22. Faculty Observation / Rating Skills Research settings –Poor inter-rater reliability –Brief rater training methods ineffective Didactic instructions Demonstration videos without practice –Accuracy: structured > open-ended forms –Increased accuracy  discriminative ability Kalet, Herbers, Noel, Kroboth

23. Faculty as Raters – Key Issues Faculty do not observe actual performance Faculty ratings lack: –Reliability –Accuracy/validity Content specificity – How comfortable are you with own skills?

24. Improving Faculty Ratings: Solutions Step 1: Getting faculty to observe –Required a part of Foundation Program –Focused observations are logistically possible –5 to 10 minute observations are valuable –Build on faculty “epiphany” The “you will not believe what I saw today” experience –Provide “usable tool”

25. Foundation Mini - CEX Tool Simple rating scale using 6 dimensions and overall rating “Structured” approach to direct observation Direct assessment of actual patient care Incorporation of CEX into daily activities Evidenced-based

26. Research: Mini - CEX Tool Two large scale U.S. studies involving 36 total residency programs Logistically feasible to incorporate miniCEX into daily activities High satisfaction among house staff Good to excellent reliability characteristics Overall scores and interpersonal scores correlated with trainee’s ECFMG OSCE scores

27. Logistics: Outpatient Clinic One mini-CEX per trainee per day per week –One attending observes portion of first visit of the day –Minimizes disruption of clinic –Perform over course of academic year –Easy to obtain 6-8 Mini-CEX’s per year per trainee in single setting

28. The Patient Encounter Sampling “parts” of the encounter: INTERVIEW PHYSICAL EXAM COUNSELING

29. Solutions: Step 2 Improve reliability –Multiple brief observations –Perform over time: outpatient setting allows for longitudinal observation –Involve multiple faculty –MiniCEX: sufficient reliability for pass/fail determinations after just 4 observations

30. Solutions: Step 3 Improve accuracy and validity –Most difficult step –Use structured rating forms –Rater training (faculty development) Caveat: brief “one time” interventions do not work

31. Does Faculty Training Work? Performance Appraisal Literature: Can reduce rating errors Can improve discriminative ability Can improve accuracy

32. Approaches to Rater Training Behavioral Observation Training Performance Dimension Training Frame of Reference Training Direct Observation of Competence Training

33. Videotape Exercise: BOT Situation: An attending is performing a miniCEX of a house officer performing a physical exam. Questions: –How well did this attending evaluate the house officer? –How was the house officer-patient interaction affected?

34. Basic Faculty Observation Skills Prepare for the observation –Faculty: Know what you’re looking for –Resident: Let them know what to expect –Patient: Let them know why you are there Minimize intrusiveness – correct positioning Minimize interference with the house officer- patient interaction Avoid distractions

35. Triangulation DESK Resident Patient Attending

36. Basic Observation Strategies Increase the amount of “sampling” –More observations lead to more accurate evaluations (“practice makes perfect”) Use of observational “aides” –Behavioral diary to record observed performance. –U.S. study: simple 3X5 card diary lead to increased comments on forms

37. Performance Dimension Training Group exercises designed to familiarize faculty with the specific elements of a competency Should involve discussion of the criteria required for each element Use defined, agreed upon elements of a competency to calibrate faculty –Playing from the “same sheet of music”

38. PDT Exercise In your small group, discuss what should be the basic components of an effective medical interview for a foundation trainee performing an outpatient consultation

39. Frame of Reference Training Goal is to improve “judgment” and accuracy Steps in FOR training: 1. Group performance dimension training (PDT) exercise 2. Review clinical vignettes that describe critical incidents of performance: unsatisfactory to average to superior

40. Frame of Reference Training 3. Faculty, using framework developed in PDT exercise, provide ratings on a behaviorally anchored rating scale (BARS) 4. Session trainer provides feedback on what “true” ratings should be for each vignette along with rationale 5. Group finishes by discussing discrepancies between trainer’s ratings and the participants’ ratings

41. Frame of Reference Training Most difficult aspect of FOR: –Setting the actual standards that distinguish between levels of performance –Reaching agreement and/or consensus among teaching faculty

42. DOC Training Combination of: –Behavioral observation training –Performance dimension training –Frame of reference training –“Live” practice in observation with standardized residents/patients Individual evaluation and feedback Group debrief with Eval and FB

43. DOC Training Trial Randomized controlled trial of 40 faculty from 16 residency programs DOC training: –High satisfaction (favorite aspect of course) –Increased comfort in observation –Changed rating behavior at 8 months –Increased accuracy in identifying unsatisfactory performance

44. Direct Observation: Challenges Like all skills, requires training and practice Faculty “calibration” important –Agreeing on “metrics” of performance –Faculty comfort with own skills Faculty training –Brief interventions mostly ineffective

45. Observation: Helpful Hints Sample “parts” of the visit: –History-taking –Physical examination –Counseling Perform longitudinally –No need to do it all at once Agree on performance metrics with faculty

46. Summary Basic clinical skills are important: so is the need to observe them! Observation is a complex skill that requires training and practice Direct observation by educators will remain a critical component of both evaluation and feedback