1. Eric Holmboe, MD Employment: ACGME Discussion of off-label drug use: not applicable 56 th ASH Annual Meeting Disclosure Statement

2. Accreditation Council for Graduate Medical Education © 2014 Accreditation Council for Graduate Medical Education What Constitutes “Good” Assessment and Milestones-based Assessment

3. The Assessment “System” Assessments within Program: Direct observations Audit and performance data Multi-source FB Simulation ITExam Qual/Quant “Data” Synthesis: Committee Residents Faculty, PDs and others Milestones and EPAs as Guiding Framework and Blueprint Accreditation Unit of Analysis: Program Certification and Credentialing Unit of Analysis: Individual JUDGEMENTJUDGEMENT D FB DD PUBLICPUBLIC

4. Process vs.Outcome Approach Educational Program VariableStructure/ProcessCompetency-based Driving force: processTeacherLearner Path of learningHierarchical (Teacher→student) Non-hierarchical (Teacher↔student) Responsibility: contentTeacherStudent and Teacher Goal of educ. encounter Knowledge acquisitionKnowledge application Typical assessment toolSingle subject measureMultiple objective measures Assessment toolProxyAuthentic (mimics real tasks of profession) Setting for evaluationRemoved (gestalt)Direct observation EvaluationNorm-referencedCriterion-referenced Timing of assessmentEmphasis on summativeEmphasis on formative Program completionFixed timeVariable time Carracchio, et al. 2002.

5. Measurement Tools: Criteria Cees van der Vleuten’s utility index: Utility = V x R x A x EI x CE/Context* Where: V = validity R = reliability A = acceptability E = educational impact C = cost effectiveness *Context = ∑ Clinical Microsystems

6. Criteria for “Good” Assessment 1 Validity or Coherence Reproducibility or Consistency Equivalence Feasibility Educational effect Learning that occurs in preparation for an assessment (e.g. certification exam; MRCP) Catalytic effect Assessment resulting in feedback that “drives future learning forward.” Acceptability 1 Ottawa Conference Working Group 2010

7. Assessing for the Desired Outcome Knows (knowledge) Knows How (competence) Shows How (performance) Does (action) Diagnostic Reasoning using clinical vignettes or CSR Performance in Practice//Multi-source feedback/ Direct Observation Standardized Patients/Simulation Multiple choice Questions Work-based assessment is mostly accomplished through the observations and questions of faculty, team members, peers and other co-workers

8. © 2014 Accreditation Council for Graduate Medical Education Milestones and Work-based Assessment  The predominant approach in CBME is work- based assessment  Milestones can serve as a framework for shared mental models and to inform observation and assessment by faculty