1. Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst, E. Holmboe

2. 2 Medical Education Trend 2000- present Competency Based Education Variable length, defined outcome Fixed length, variable outcome Structure/Process Knowledge acquisition Single subjective measure Norm referenced evaluation Evaluation setting removed Emphasis on summative Competency Based Knowledge application Multiple objective measures Criterion referenced Evaluation setting: DO Emphasis on formative Caraccio et al 2002

3. 3 In-Training Performance Assessment  Assessment in authentic situations  Learners’ ability to combine knowledge,skills, judgments, attitudes in dealing with realistic problems of professional practice  Assessment in day to day practice  Enables assessment of a range of essential competencies, some of which cannot be validly assessed otherwise Govaerts MJB et al. Adv Health Sci Edu. 2007;12:239-60

4. 4 Observation to Test Assumptions  Direct observation can test assumptions  4 observations needed to detect outliers  Shared responsibility Detect OutliersFeedback/development TIME/TASK Early Late

5. 5 Observation and Safe Patient Care  Importance of appropriate supervision  Entrustment Trainee performance* X Appropriate level of supervision** Must = Safe, effective patient-centered care * a function of level of competence in context **a function of attending competence in context

6. 6 Types of Supervision  Routine oversight  Clinical oversight planned in advance (i.e. what we normally do)  Responsive oversight:  Clinical activities that occur in response to trainee or patient specific issues (i.e. you do more than usual)  Direct patient care:  When supervisor moves beyond oversight to actively providing care for the patient  Backstage oversight:  Clinical oversight which the trainee is not aware of Kennedy TJT et al. JGIM 2007.22:1080-85.

7. 7 Your Supervision  How do you usually supervise?  When do you supervise more closely?  How do you change your supervision to ensure patients get safe, effective, patient-centered care?  What did you learn observing that will change how you supervise going forward?  REMEMBER: SUPERVISION ALSO FOR FEEDBACK

8. 8 Entrustment  “A practitioner has demonstrated the necessary knowledge, skills, and attitudes to be trusted to independently perform this activity.” Ten Cate O, Scheele F. Acad Med 2007;82:542-7

9. 9 Problems with Performance Assessment  Poor accuracy  Focus on different aspects of clinical performance  Differing expectations about levels of acceptable clinical performance  Rating errors  Halo effect/ “Horn” effect  Leniency/stringency effect

10. 10 Factors That May Impact Ratings  Minimal impact of demographics  Age, gender, clinical and teaching experience  Faculty’s own clinical skills may matter  Faculty with higher history and patient satisfaction performance scores provide more stringent ratings. Kogan JR. et al. Acad Med. 2010;85(10 Suppl):S25-8

11. 11 Factors Influencing Faculty Ratings  Different frameworks for judgments/ratings  Self-as-reference (predominant)  Trainee level  Absolute standard  Practicing physicians

12. 12 Faculty OSCE Clinical Skills CompetencyMean (SD)RangeGeneraliz- ability History Taking65.5% (9.6%)34% - 79%0.80 Physical Exam78.9% (13.6%) 36% - 100%0.52 Counseling77.1% (7.8%)60% - 93%0.33 Patient Satisfaction 1 5.62 (0.48)4.43 – 6.630.60 1 On 7-point scale Kogan JR. et al. Acad Med. 2010;85(10 Suppl):S25-8 N=44

13. 13 Other Factors Influencing Ratings  Factors external to resident performance  Encounter complexity  Resident characteristics  Institutional culture  Emotional impact of constructive feedback  Role of inference

14. 14 Definition of Inference a. The act or process of deriving logical conclusions from premises known or assumed to be true. b. The act of reasoning from factual knowledge or evidence.

15. 15 Affix meaning Concrete data (resident actions) Conclusions Assumptions 1.1. 2.2. 3.3.

16. 16 The Problem with Inference  Inferences are not recognized  Inferences are rarely validated for accuracy  Inferences can be wrong

17. 17 Types of Inference about Residents  Skills  Knowledge  Competence  Work-ethic  Prior experiences  Familiarity with scenario  Feelings  Comfort  Confidence  Intentions  Ownership  Personality  Culture

18. 18 High Level Inference

19. 19 Direct Observation: A Conceptual Model Kogan JR, et al. Med Educ. 2011

20. 20 International Comparative Work  Yeates (UK)  Differential salience  Criterion uncertainty  Information integration  Govaerts (Netherlands)  Use of task-specific and person schemas  Substantial inference in person schema  Rater idiosyncrasy  Gingerich (Canada)  Impact of social models: clusters; person; labels

21. 21 Achieving Accurate, Reliable Ratings  Form is not the magic bullet  Assessment requires faculty training  Similar frameworks  Agreed upon levels of competence  Move to criterion referenced assessment

22. 22 Questions