Building assessment capacity in medical education Hamish Coates

that identifies key indicators of educational quality – the things that really count sets externally referenced and context-relevant standards of performance collects quantitative data on performance uses that data to highlight areas of strength and improve areas of weakness provides information to potential students in an informative and inspiring fashion assures the public that minimum standards of performance are being met Imagine a school…

An experiment 95%75%

Score distributions for eight groups (institutions, faculties, subjects, etc.) “Standards”, not “standardisation” Performance Minimum standard

Robert Rauschenberg, Untitled, c1951

We need to decide what ‘standards’ means “One might postulate that there are competing monologues in non-intersecting idiolects and/or sociolects and that nobody does actually fully understand anybody else when these terms are used.” (Alexander, 2009) ‘Standards’ – the what: (indicators), eg: teaching quality, learning outcomes, etc… ‘Standards’ – the how: (performance / level): eg: high, low, competent, thresholds, criteria, etc… (“Standards have fallen over the last 20 years.”)  ‘Standards’ ~ ‘levels of performance on educational indicators’

Standards The Profession Graduate Outcomes PBF TESQA Universality Compacts Distributed education Competition / marketing / rankings Curriculum and learner diversification Morality/safety Shaping contexts

Conceptually sound Workable Cost effective Protect academic autonomy Reflect and stimulate diversity Inspires excellence Auditable Technically robust Beyond boutique Improvement focused Support risk-based monitoring Provide public information Public and private Quantitative focus Outcomes focused Learning focused Multilevel responsibility Let’s improve Assessment design principles

Assessment framework: core areas; divergent topics Items sourced from academics, existing tests, other systems… Items quality assured, criterion referenced, and calibrated Items mapped to measure ‘core content’ Items designed to measure ‘above content’ (cf: AHELO) Items deployed to monitor ‘minimum standards’ Cross-institutional/-national linkages An efficient, sustainable collaboration Objective data for evaluating curriculum and performance MCA

Assessment framework: core areas; divergent topics Library of catalogued items built via faculty training workshops Items quality assured, criterion referenced, and calibrated ‘Core’ set of items deployed to monitor ‘minimum standards’ Sampled items embedded into routine assessment Return performance data for calibration Cross-institutional/-national linkages An efficient, sustainable collaboration Objective data for evaluating curriculum and performance AMAC

Training Collaborating Constructing Awareness Reflecting Replication Theorising Designing Eight engagement opportunities Eight engagement opportunities

Building assessment capacity in medical education Hamish Coates