 Mennin Consulting, 2006 Overview of Assessment ESME 2006 Stewart Mennin, PhD

 Mennin Consulting, 2006 ESME Objectives Make more informed decisions about assessment Apply information to your teaching Describe the relationship between assessment & learning Discuss criteria for effective assessment Define & apply basic terminology

 Mennin Consulting, 2006 ESME Objectives Describe selection, supply & performance methods Link Miller’s pyramid to assessment methods Match assessment methods to your outcomes Draft assessment plan (blueprint)for your teaching

 Mennin Consulting, 2006 ESME GOALS OF ASSESSMENT Enhance student learning Guide faculty teaching efforts Inform curricular development & revision Fulfill responsibilities to students, institution & society Contribute to research

 Mennin Consulting, 2006 ESME FEATURES OF ASSESSMENT Assessment is more a curriculum design problem than a measurement problem Part of learning & can not be separate from it Assessment is about the present, past and future of learning Assessment in not a collection of instruments Assessment should be authentic, grounded in what is real

 Mennin Consulting, 2006 ESME EIGHT CRITERIA: EFFECTIVE ASSESSMENT 1. Articulation of a clear conception of intended learning outcomes 2. Use variety of assessment procedures/methods 3. Close agreement between intended outcomes of instruction, learning tasks, & the assessment procedures/methods 4. Adequate sampling of student performance Gronlund, 2000

 Mennin Consulting, 2006 ESME EIGHT CRITERIA: EFFECTIVE ASSESSMENT 5. Procedures that are equitable for all participants 6. Explicit, specific criteria for use in judging successful performance 7. Timely feedback that emphasizes strengths of performance & focuses attention on specific areas in need of improvement 8. A grading & reporting system consistent with the rest of the learning & assessment process Gronlund, 2000

 Mennin Consulting, 2006 ESME Key Terminology Formative – Summative Reliability & Validity Criterion Referenced --- Norm referenced Progress test Sequential test

 Mennin Consulting, 2006 ESME Assign grades Judge competence Certify mastery Promotion Feedback to student & learner Find misconceptions Identify strengths & weaknesses Change future teaching Develop learning plan Teacher Teacher & learner Milestones & endFrequent & throughout Decisions / JudgmentImprovement Use Evaluator Timing Purpose Summative Formative Assessment As Learning

 Mennin Consulting, 2006 ESME Reliability Consistency of results Reliability is matter of sampling – Across content

 Mennin Consulting, 2006 ESME Domain of Interest Test Sample 

 Mennin Consulting, 2006 ESME Assessment of Clinical Competence is a Issue of Specificity Competent

 Mennin Consulting, 2006 ESME Reliability Practical suggestions: – Do not rely on short tests – Sample broadly (content, in time, examiners, patients) – Consider efficiency in: selection of test format construction test items

 Mennin Consulting, 2006 ESME Reliability Adequate reliability requires substantial sampling (therefore: resources, testing time, examiners, patients, etc.) – Be efficient

 Mennin Consulting, 2006 ESME Validity Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7. Knows Shows how Knows how Does Professional authenticity

 Mennin Consulting, 2006 ESME Validity Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7. Knows Shows how Knows how Does Professional authenticity Written, Oral or Computer based assessment Performance or hands on assessment

 Mennin Consulting, 2006 ESME Valid Inferences Validity is a single concept with different forms of evidence – Content-related evidence – Criterion-related evidence – Construct-related evidence

 Mennin Consulting, 2006 ESME ASSESSMENT METHODS All assessment methods have limitations and no one method can assess all types of knowledge & skill Multiple methods are recommended

 Mennin Consulting, 2006 ESME Some Assessment Options: Selection Item – Multiple Choice Question – One-Best Answer Questions – Extended-Matching Questions Supply Items – Free Response Questions – Short Answer Questions – Essay Questions Performance Examinations – Laboratory Practical Examinations (Identification and Concepts) – Clinical Skills (e.g., OSCE) – Other

 Mennin Consulting, 2006 ESME Selection Methods MCQ Matching Extended Matching True False Multiple True False Knows Knows howShows how Does

 Mennin Consulting, 2006 ESME Knows/Knows how The stimulus format is more important than the response format Knows Knows how Shows how Does

 Mennin Consulting, 2006 ESME How Well Written? Stem a.Poor b.Good

 Mennin Consulting, 2006 ESME How Well Written? a. Stem b. c. a. Poor b. Good d.

 Mennin Consulting, 2006 ESME How Well Written? 30-year-old man has loss of pain and temperature sensation from the neck down on the right side of the body and on the left side of the face; partial paralysis of the soft palate, larynx, and pharynx on the left; and ataxia on the left. This syndrome is most likely to result from thrombosis of which of the following arteries? ABasilar BRight posterior inferior cerebellar CLeft posterior inferior cerebellar DRight superior cerebellar ELeft superior cerebellar

 Mennin Consulting, 2006 ESME Extended Matching For each quotation, select the most likely author. 1.I never teach my pupils. I only attempt to provide the conditions under which they can learn. 2.All genuine knowledge originates in direct experience. 3.To know one’s ignorance is the best part of knowledge. Authors on Education: a.D. J. Boorstene.Lao-Tzu b.Camusf.S. Scott Obenshain c.Albert Einsteing.John Pieper d.Thomas Harrish.Mao Tse-Tung

 Mennin Consulting, 2006 ESME Extended Matching Answers Authors on Education: a.D. J. Boorstene.Lao-Tzu b.Camusf.S. Scott Obenshain c.Albert Einsteing.John Pieper d.Thomas Harrish.Mao Tse-Tung 1. I never teach my pupils. I only attempt to provide the conditions under which they can learn. (c) 2. All genuine knowledge originates in direct experience. (h) 3. To know one’s ignorance is the best part of knowledge. (e)

 Mennin Consulting, 2006 ESME Supply Methods MEQ Essay (long & short) Practical Short Answer Written simulations Knows Knows how Shows how Does

 Mennin Consulting, 2006 ESME Performance Assessment Procedures – skills lab OSCE (with skills stations) Mini CEX Long Case Logs Portfolio Clinical Notes Observation Other Knows Knows how Shows how Does

 Mennin Consulting, 2006 ESME Communication Skills Interviewing & Communication Skills

 Mennin Consulting, 2006 ESME Essential Elements of Communication Skills Open the Discussion Build a Relationship Gather Information Understand the Patient’s Perspective Share Information Reach Agreement Provide Closure Acad Med 2001;76(4):

ClinicalProblem

Key Feature Approach Testing time Case 1 Case 2 Test using integral cases Case 1 Case 2 Case 3 Case 4 Case 5 Case 6 Case 7 Test using key features

 Mennin Consulting, 2006 ESME Shows How: OSCE Make stations as clinically authentic as possible Global judgments do well in OSCEs Content specificity is the problem Content specificity Knows Knows how Shows how Does

 Mennin Consulting, 2006 ESME Checklist/Global Rating Scale Comparison Checklists – Dichotomous scores (yes/no) – Relatively objective Record rather than interpret – Station specific Rating Scales – Ordinal level measures (poor, good, excellent) – Relatively subjective – Station independent

 Mennin Consulting, 2006 ESME Does: Practice Assessment Indirect methods – Chart audit – Practice audits – Prescription data……… Direct methods – Clinical (supervisor) ratings – Undercover standardized patients – Video assessment – Patient reports – Peer reports – Clinical work samples – Mini-CEX……… Knows Knows how Shows how Does

 Mennin Consulting, 2006 ESME Clinical Work Samples Repeated direct observations of clinical performance in practice using (generic) evaluation forms, completed by any significant observer (clinician, nurse, peer…..) Illustration Knows Knows how Shows how Does

 Mennin Consulting, 2006 ESME Mini Clinical Examination (Norcini, 1995) Short observation during clinical patient contact (10-20 minutes) Oral evaluation Generic evaluation forms completed Repeated at least 4 times by different examiners (cf. Knows Knows how Shows how Does

 Mennin Consulting, 2006 ESME NEW SKILLS EMPHASIZED Learning how to learn Self-assessment Humanization Leadership Team skills Skills of expression Professional behavior Reflection Permanent education

 Mennin Consulting, 2006 ESME Methods & Knowledge

 Mennin Consulting, 2006 ESME Task Use the table from effective teaching and match assessment methods to your outcome competencies

 Mennin Consulting, 2006 ESME Progress Test Scores on one progress test Faculty of Medicine Maastricht University

 Mennin Consulting, 2006 ESME Mean correct scores on the Progress Test from a number of Maastricht student cohorts (black lines) and reference group of family physicians in training (blue line) Progress Test Maastricht Test Results ‘77 - ‘88 Faculty of Medicine Maastricht University

 Mennin Consulting, 2006 ESME Planning: Blueprinting Competencies Time Teaching methods In class -- out of class

Competencies Perform-2 Capable Perform-1 Competent Supply Select Methods Clinical Skills Critical Int Know Comm Human Self Assess Profess

 Mennin Consulting, 2006 ESME Conclusions Assessment is less of a psychometric problem, but rather an educational design problem (i.e. how to use assessment strategically for its educational effects) Assessment requires careful planning and monitoring To effective with the entire competence pyramid – a mix of methods that depends on the context; not a single method is best or bad Cees van der Vleuten

 Mennin Consulting, 2006 ESME Acknowledgements Cees van der Vleuten - Maastrict Deana Richter – New Mexico Teresita McCarty - New Mexico Nancy Sinclair - New Mexico Regina Petroni Mennin – UNIFESP

 Mennin Consulting, 2006 ESME Presentations Available at: