1. How Behavioral Science Faculty Can Help Competency Assessment Glow and Flow Larry Mauksch, M.Ed Senior lecturer, Department of Family Medicine University of Washington School of Medicine Incoming Editor, Families, Systems, and Health Presentation and video will be placed on FMDRL Larry Mauksch, M.Ed University of Washington Department of Family Medicine

2. Disclosures  Grant support from the Picker Foundation and Arthur Vining Davis Foundations  Paid private consultant to academic and community health organizations  Paid by STFM as an “On the Road” faculty  Co-owner of a formative assessment and development software company, MedFAD.com Larry Mauksch, M.Ed University of Washington Department of Family Medicine

3. Outline Background: NAS, CBE, Milestones, EPAs Understand the differences and connections between formative and summative assessment Direct observation is criticalIt is about you, not the toolPromote tacit knowledge articulationDissolve evaluation PTSDSupport cultural change

4. Half Yearly Assessment Cycle Data collection and consolidation from multiple sources Given to: Resident Advisor CCC CCC provides summary impression ( with updated milestone rating) to resident and advisor Resident and Advisor meet to review summaries, CCC impressions and create IEP CCC summary and IEP to faculty influencing subsequent formative assessment

5. Milestones and Entrustable Professional Activities (EPAs) Milestones Read them? Used them?

6. Patient Care and procedures PC-1 Urgent and Emergent PC-2 Chronic illness PC-3 Health prevention promotion PC-4 Undiffer- entiated patients PC-5 Proced- ures Medical knowledge MK-1 breadth and depth MK-2 Critical thinking Professionalism Prof-1 Fully integrate Prof-2 Action awareness Prof-3 Humanism Cultural Prof-4 Self care growth Systems based practice SBP-1 Cost conscious SBP-2 Safety SBP-3 Advocate SBP-4 Coordin’te team care Practice based learning and improvement PBLI-1 Find / use evidence PBLI-2 Self direct’d learning PBLI-3 Improves system Communication C-1 Relation- ships C-2 Effective C-3 With team, others C-4 Improve w/ tech Family Medicine: 6 Competencies and 22 Milestones

7. Milestone Levels  Level 1: The resident demonstrates milestones expected of a resident who has had some education in family medicine.  Level 2: The resident is advancing and demonstrating additional milestones.  Level 3: The resident continues to advance and demonstrate additional milestones ; the resident consistently demonstrates the majority of milestones targeted for residency.  Level 4: The resident has advanced so that he or she now substantially demonstrates the milestones targeted for residency. This level is designed as the graduation target.  Level 5: The resident has advanced beyond performance targets set for residency and is demonstrating “aspirational” goals which might describe the performance of someone who has been in practice for several years. It is expected that only a few exceptional residents will reach this level.

8. Henry, Holmboe, Frankel. Evidence-based competencies for improving communication skills in graduate medical education: A review with suggestions for implementation. Med Teach. May 2013;35(5):395-403. Competency— ” ability to… Take accurate and complete patient historiesPatient Care 3, level 1 Communicate with other doctorsComm 3, L 1-5: SBP 4, L 3 Communicate w/ other health care team members Comm 3, L 1-5; SPB 4, L 3 Set agendasCommunication 2, Level 3 Assess and improve patient adherencePatient Care 2, Level 2-5 Deliver diagnostic and prognostic newsCommunication 2, Level 2,4 Elicit patients’ beliefs, perspectives, concernsC 1, L 2; C 2, L 3; Prof 3, L 2 Treatment plansPC 1, L 2-3; PC 2 L 2-3; PC 4, L 4; Prof 3, L 3 Establish rapport and demonstrate empathyC 1 L 2-4; PC 4, L4; Prof 3, L 1 Manage conflict and negotiate with patientsCommunication 1, Level 4 Basic patient counseling skillsCommunication 2, Level 4 Counseling families and caregiversCommunication 2, Level 4

9. Enstrustable Professional Activities Olle ten Cate, PhD Journal of Graduate Medical Education, March 2013 “Tasks or responsibilities that can be entrusted to a trainee once sufficient, specific competence is reached to allow for unsupervised execution” Larry Mauksch, M.Ed University of Washington Department of Family Medicine

10. Entrustable Professional Activities and the Milestones Larry Mauksch, M.Ed University of Washington Department of Family Medicine EPA: Newborn care Common mental illness Chronic illness in adults Health behavior change Office Procedures End of life and palliative care IPCProfPt CareSBPPBLIMK

11. Both are Necessary and Neither are Sufficient for CBE EPAs Milestones Skills Knowledge Attitudes Larry Mauksch, M.Ed University of Washington Department of Family Medicine

12. Which Statements are true? 1. Milestones are designed for reporting resident development to the ACGME 2. Milestones can help guide your teaching of residents throughout training 3. It is expected that a mature family medicine curriculum will produce most graduating residents at level 5 for all milestones 4. The milestones represents years of educational research about professional development in family medicine Larry Mauksch, M.Ed University of Washington Department of Family Medicine

13. Which Statements are true? 1. Milestones are designed for reporting resident development to the ACGME 2. Milestones can help guide teaching of residents throughout training 3. It is expected that a mature family medicine curriculum will produce most graduating residents at level 5 4. The milestones represents years of educational research about professional development in family medicine Larry Mauksch, M.Ed University of Washington Department of Family Medicine

14. Miller’s Assessment Pyramid Faculty observation (with real patients) Does Standardized patients Shows how Matching or critical response questioning Knows how Multiple choice questions Knows Impact on the patient Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med 65(9 Suppl): S63-7. Adapted from Holmboe and Hawkins. Evaluation of Clinical Competence, Mosby 2008 14

15. Time to Achieve Reliability Across Tools Medical Education 2005; 39 : 309–317 (references to studies omitted but are in the article) InstrumentDescription1 hr2 hrs4hrs8hrs Multiple ChoiceShort stem.62.76.93,93 Patient management prob Simulation of patient.36.53.69.82 Key feature caseShort pt case, followed by write in answer.32.49.66.79 Oral examinationOral exam on pt cases.50.69.82.90 Long case examOral exam on previously unobserved real patient.60.75.86.90 OSCESimulate realistic encounters, round robin.54.69.82.90 Mini- CEXShort follow up oral exam on observed real patient.73.84.92.96 Practice video assessment Selected encounters from real practice.62.76.93 Incognito SPReal consults scored by undetected SPs.61.76.82.86

16. Miller’s Assessment Pyramid Contextual assessment Entrustable professional activities Qualitative measure relying on faculty expertise Does Discrete KSAs Standardized Discrete tools Shows how Knows How Knows Impact on the patient Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med 65(9 Suppl): S63-7. Van Vleuten and Schuwirth. Assessing professional competence: from methods to program. Medical Education, 2005, 309-17 16

17. Understand the Differences and Connections between Formative and Summative Assessment Larry Mauksch, M.Ed University of Washington Department of Family Medicine

18. Formative vs Summative Assessment to Meet ACGME Requirements 1. Formative assessment is not necessary if you have a high quality summative assessment program 2. A high quality summative assessment is a function of high quality formative assessment Larry Mauksch, M.Ed University of Washington Department of Family Medicine

19. Model of Program Assessment Van Der Vleuten et al. Medical Teacher 2012; 34: 205–214 Principles of Assessment Any single data point is flawed Any standardized assessment can have validity built into the instrument Validity of non-standardized assessment resides in the users and not so much in the instruments The number of data points used should be on a continuum and should be proportional to the stakes of the assessment Assessment drives learning by providing meaningful information to the learner Expert judgment is imperative

20. It is more about you than the tool van der Vleuten et al The assessment of professional competence: building blocks for theory development. Best Pract Res Clin Obstet Gynaecol. Dec 2010;24(6):703-719 Trained faculty offer formative assessment influenced by context and focused on need in ways that checklists can not provide Multiple observations with purposeful sampling by multiple expert raters offering authentic assessments may provide a more valid picture of “does” than quantitative tools

21. Conflict of Interest Issues: Formative and Summative Mindsets Can faculty functioning in a formative mode, fairly block out summative thinking? Should faculty on the CCC provide formative assessment? How much involvement should residents have in creating formative notes?

22. Don’t allow quantitative reductionism caused by fulfilling ACGME reporting requirements to prevent you from providing meaningful summative statements to learners Larry Mauksch, M.Ed University of Washington Department of Family Medicine

23. Direct Observation is Critical Ross V, Mauksch L, Huntington J, Beard JM. Interdisciplinary direct observation: impact on precepting, residents, and faculty. Fam Med. May 2012;44(5):318- 324.

24. ‘Documented observations with feedback is the sine qua non of competency based education’ M. Donoff, MD University of Alberta Donoff MG. Field notes: assisting achievement and documenting competence. Can Fam Physician. Dec 2009;55(12):1260-1262, e1100-1262. 24

25. Risks and Benefits of Checklists Which statements are false? 1. Using multiple instruments to assess multiple competencies may overwhelm programs because reliable sampling requires 10 -14 observations 2. They ignore context 3. Verbal anchors are better than numbers 4. They help teach observers as well as trainees 5. They help teach, reinforce and assess critical or broadly applied skills 6. Adding subjective narrative to checklists increases assessment value

26. Faculty Development Ineffable– we can know more than we tell The Tacit Dimension (1966) Michael Polanyi, MD, PhD Our challenge is to unpack, deconstruct, articulate As we do this our trainees grow and we grow as clinicians and teachers

27. Teachers as Learners Wenrich MD, Jackson MB, Ajam KS, Wolfhagen IH, Ramsey PG, Scherpbier AJ. Teachers as learners: the effect of bedside teaching on the clinical skills of clinician-teachers. Acad Med. Jul 2011;86(7):846-852. Egnew TR, Mauksch LB, Greer T, Farber SJ. Integrating communication training into a required family medicine clerkship. Acad Med. Aug 2004;79(8):737-743. Mauksch L, Farber S, Greer HT. Design, Dissemination, and Evaluation of an Advanced Communication Elective at Seven U.S. Medical Schools. Acad Med. Apr 29 2013. Beckman TJ. Lessons learned from a peer review of bedside teaching. Acad Med. Apr 2004;79(4):343-346.

28. Deconstructing by Creating SCRIPPT Scratch your head What is the problem? What did I see? How would I say or do this? Create an explanation, verbal example (out loud) or an action Interpret your understanding within a KSA category Practice --Have the trainee try or Praise to enhance awareness of skill use Troubleshoot- Consider pitfalls and refinements

29. Show first video- poor closing- - here

30. Deconstructing by Creating SCRIPPT Scratch your head What is the problem? What did I see? How would I say or do this? Create an explanation, verbal example (out loud) or an action Interpret your understanding within a KSA category Practice --Have the trainee try or Praise to enhance awareness of skill use Troubleshoot- Consider pitfalls and refinements

31. Show better video here- combining AVS and Teachback

32. Deconstructing by Creating SCRIPPT Scratch your head What is the problem? What did I see? How would I say or do this? Create an explanation, verbal example (out loud) or an action Interpret your understanding within a KSA category Practice --Have the trainee try or Praise to enhance awareness of skill use Troubleshoot- Consider pitfalls and refinements

33. Evaluation PTSD Larry Mauksch, M.Ed University of Washington Department of Family Medicine

34. Evaluation PTSD Delayed, negative, often summative feedback with no formative context Shame and belittlement. Often experienced or observed in the presence of others Faculty may not admit that some knowledge is ineffable and resort to “pimping” Produces An unconscious barrier to providing helpful and supportive feedback or behavior that is replicated in one’s own teaching

35. Competency Based Education Implements and evaluates CBE in residency program Does Demonstrates use of direct observation feedback to learner Shows how Can describe and generate competency assessment tools/skills Knows how Can define formative and summative assessment Knows Impact onTraining Adapted from Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med65(9 Suppl): S63-7. 35

36. Behavioral Science Faculty Can Lead Manage your anxiety about the milestones. They help you know where to aim your teaching and deconstruction Remember, its about faculty development, not the toolsRole model deconstructing tacit knowledgeAddress evaluation PTSDLead with tracked, organized, formative assessment Larry Mauksch, M.Ed University of Washington Department of Family Medicine

37. ACGME website Webinar #3 Implementing Milestones, Clinical Competency Committees and Assessment, second section, by Pamela Derstine, PhD “ Assessment of Resident Milestones” Donoff MG. Field notes: assisting achievement and documenting competence. Can Fam Physician. Dec 2009;55(12):1260-1262, e1100-1262. Ross S, Poth CA, Donoff MG, et al. Involving users in the refinement of the competency-based achievement system: an innovative approach to competency- based assessment. Med Teach. 2012;34(2):e143-147. Ross S, Poth CN, Donoff M, et al. Competency-based achievement system: using formative feedback to teach and assess family medicine residents' skills. Can Fam Physician. Sep 2011;57(9):e323-330. Dijkstra, Van Der Vleuten, Schuwirth A new framework for designing programs of assessment, Adv in Health Sci Ed 2010 15, 379-393 Miller, G. E. The assessment of clinical skills/competence/performance. 1990 Acad Med 65(9 Suppl): S63-7. Competence Based Education: Selected articles and resources

38. Larry Mauksch, M.Ed University of Washington Department of Family Medicine Driessen EW, van Tartwijk J, Govaerts M, Teunissen P, van der Vleuten CP. The use of programmatic assessment in the clinical workplace: a Maastricht case report. Med Teach. 2012;34(3):226-231. Holmboe ES, Ward DS, Reznick RK, et al. Faculty development in assessment: the missing link in competency-based medical education. Acad Med. Apr 2011;86(4):460-467. Henry SG, Holmboe ES, Frankel RM. Evidence-based competencies for improving communication skills in graduate medical education: A review with suggestions for implementation. Med Teach. May 2013;35(5):395-403. Schuwirth LW, Van der Vleuten CP. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach. 2011;33(6):478-485. van der Vleuten CP, Schuwirth LW. Assessing professional competence: from methods to programmes. Med Educ. Mar 2005;39(3):309-317. van der Vleuten CP, Schuwirth LW, Driessen EW, et al. A model for programmatic assessment fit for purpose. Med Teach. 2012;34(3):205-214. van der Vleuten CP, Schuwirth LW, Scheele F, Driessen EW, Hodges B. The assessment of professional competence: building blocks for theory development. Best Pract Res Clin Obstet Gynaecol. Dec 2010;24(6):703-719.

39. EPA articles  March, 2013 Issue of the Journal of Graduate Medical Education  Carraccio CL, Englander R. From flexner to competencies: reflections on a decade and the journey ahead. Acad Med. Aug 2013;88(8):1067-1073.  ten Cate O, Snell L, Carraccio C. Medical competence: the interplay between individual ability and the health care environment. Med Teach. 2010;32(8):669-675.  Chang A, Bowen JL, Buranosky RA, et al. Transforming primary care training--patient-centered medical home entrustable professional activities for internal medicine residents. J Gen Intern Med. Jun 2012;28(6):801-809. Larry Mauksch, M.Ed University of Washington Department of Family Medicine

40. Thank you Questions and Comments? Larry Mauksch, M.Ed University of Washington Department of Family Medicine