Reflections on the Migration of Medical Simulators from Training Tools to Assessment Tools Rich Shavelson Stanford Education Assessment Lab Invited Talk Simulation in Medical Education Seminar Stanford Hospital March 14, 2007

Overview Motivation for Talk Analogy and Implications: Medical Simulators and Job Performance Measurement Evaluation of Technical Quality of Simulator Assessments Open Discussion

Motivation For Talk Requests for advice on how to score performance collected with medical training simulators –SUMMIT—Surgical laparoscopic simulators Creating scores from data reported by simulator Defining reference groups Defining performance benchmarks Technical quality –Pediatric airway management simulator Separating training and assessment Developing a profile of performance Task sampling Technical quality Before Ideal & Possible Solution After Imple- menting Solution

Analogy and Implications: Medical Simulators and Job Performance Measurement A great deal of work has been done on job & education performance measurement that applies to medical simulators as assessment tools. My enduring interest

Analogy Continued Approaches to performance measurement –Construct definition –Task sampling definition –Rapprochement Cold reality Universe of Performance of Interest Universe of Possible Tasks for Assessment Universe of “Do-Able” Tasks Universe of “Do-Able” Task Formats Tasks & Scoring on Test

Evaluation Of Technical Quality Reliability –Classical test theory (“Cronbach’s Alpha”)— simple task sampling –Generalizability theory—task sampling –Item response theory—construct driven Validity –Construct –Content –Predictive

Thanks… Your Turn!