Scaling Kirkpatrick's Pyramid: An Overview of Higher-level Evaluation Miguel Paniagua, MD, FACP American Geriatrics Society Meeting Orlando, FL Saturday, May 15, 2010 MOVING BEYOND LEARNER SATISFACTION: A WORKSHOP ON THE EVALUATION OF EDUCATIONAL PROJECTS AND PROGRAMS

Objective Compare and contrast the utility of different metrics for project evaluation (e.g., learner satisfaction, learner knowledge, behavior change, and project impact

Scholarship The foundation for academic promotion providing a common ground for assessing the diverse roles & contributions of faculty to the mission of the medical school -Simpson & Fincher, Acad Med, 1999 A tangible product of some new knowledge or its presentation that is peer reviewed and publicly disseminated

Kirkpatrick Levels of Evaluation 5. Patient outcomes 4.Organizational change 3. Change in behavior: taking it to the bedside 2. Measurement of learning: knowledge, skills and attitudes 1. The response to learning: satisfaction L Hutchinson, BMJ 1999;318: ( 8 May )

Number of articles according to type of primary participant and outcome, from data-driven studies reported in three leading medical education journals from 1996 to 1998 Medical Education, 2001;35:

UseIndividualProgrammatic Formative Evaluation of learners  Ensure progression towards competency Evaluation of faculty  Identify areas for improvement  Identify areas for curricular improvement  Identify opportunities for curriculum renewal & ongoing enhancement SummativeEvaluation of learners  Verify competence  Motivate continual improvement  Grades & Promotion  Judgments about success, efficacy  Decisions on allocation of resources  Motivation & Recruitment  Dissemination  Ensure sustainability

AAMC Competency ObjectivesEducational StrategiesAssessment Methods Define & Distinguish among the clinical presentations of delirium, dementia & depression List the diagnostic criteria for dementia, delirium and depression (Cognitive- Recall) Reading Lecture & discussion (i.e. the DEMENTIA curricular materials) Short answer MCQ Given a series of cases, distinguish between dementia, delirium and depression (Cognitive- Comprehension) Reading Lecture & discussion Video Review Case discussion (i.e. The “3 D’s card sort game) Short answer MCQ OSCE Perform & interpret a cognitive assessment in an older patient for whom there are concerns regarding memory or function Given a patient with suspected dementia, demonstrate a properly-administered SLUMS examination during one simulated and one actual encounter (Psychomotor, Skill) Demonstration Role Play Clinical Experience Simulation (i.e. The Case of William Harris SP) Unstructured observation Direct observation Performance audit OSCE Rank cognitive assessment as an important component of geriatric medical care (Affective) Reading Lecture/discussion Learning projects Behavioral Experiential essay Short answer