Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014

Assessment Formative Summative

Formative Assessment Assessment for learning

Summative Assessment Assessment of learning

Evaluation Methodology Bould MD, et al. Assessment of procedural skills in anesthesia. Br J of Anaesth 2009; 103:

Methodology: Global Rating Scale Overall performance by categories Scores (Likert) Drawbacks

Global Rating Scale Blum RH, et al. Simulation-based assessment to identify critical gaps in safe anesthesia resident performance. Anesthesiology 2014; 120:129-41

Methodology: Checklists Delineated actions Weighted, timed, cumulative... Adverse events Drawbacks

Checklists Schwid, et al. Evaluation of anesthesia residents using mannequin-based simulation, a multi institutional study. Anesthesiology 2002; 97:

Methodology: “Time To” Timing of critical actions Pace Motion analysis

Methodology: Survey Data Self assessments Psychometrics: Pre and Post Testing

Survey Data Lammers RL, et al. Simulation-based Assessment of Paramedic Pediatric Resuscitation Skills. Prehospital Emergency Care 2009; 13:

Methodology: Biometrics “Technometrics?” Mannequin physiology

Validity The Assessment Method The Simulation

Validity: “Face” General acceptability of the tool used for the task required Does this make sense?

Validity: “Content” Does the content match what’s to be assessed? Task Analysis The Experts

The Delphi Method BOGSAAT

Validity: “Construct” Does this reflect true performance? Experts Role of experience Surveys

Validity: “Construct” Statistical Analysis: Discrimination: “D Statistic” ANOVA “Concurrence”

Tool Validation Reid J, et al. The Simulation Team Assessment Tool (STAT): Development, reliability and validation. Resuscitation 2012; 83:

Validity: “Criterion” Or “Extrapolation” Similar results to other performance measures “Predictive”

Consistency: Internal Requires realism Reflects test quality Pilot Testing

Consistency: Statistical Analysis Do testers do well or poorly on the same items? Generalizability: “G statistic”

Reliability: Inter-Participant Set-up Simulator The Subject

Reliability: The Rater Inter-rater reliability (external reliability) Statistics: IRR, ICC

Reliability: The Rater Rater training Rater refreshing Scoring

Reliability: The Observation Real time Recording Bias

Validated Tools Cheng A, et al. Designing and Conducting Simulation-Based Research. Pediatrics 2014; 133:

Take Away Rigor in realism, rehearsing and piloting Mix Checklists and GRS Supplement with survey, psychometric, device data, etc More tasks with more (recorded) views

Further Sourcing Schwid, et al. Evaluation of anesthesia residents using mannequin-based simulation, a multi institutional study. Anesthesiology 2002; 97: Devitt, et al. The validity of performance assessment using simulation. Anesthesiology 2001; 95:36-42 Devitt, et al. Testing internal consistency and construct validity during evaluation of performance in a patient simulator. Anesth Analg 1998; 86: Devitt JH, Kurrek MM, Cohen MM, Fish K, Fish P, Murphy PM, Szalai JP: Testing the raters: Inter-rater reliability of standardized anaesthesia simulator performance. Can J Anaesth 1997; 44:924–8 Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R: Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998; 89:8-18 Boulet JR, et al. summative assessment in medicine: the promise of simulation for high-stakes evaluation. Acad Emerg Med 2008; 15: Cheng A, et al. Designing and Conducting Simulation-Based Research. Pediatrics 2014; 133: Furman GE et al. Quality Assurance Best Practices for Simulation-Based Examinations. Qual Assurance Best Practices 2014; 5: Reid J, et al. The Simulation Team Assessment Tool (STAT): Development, reliability and validation. Resuscitation 2012; 83: Siassakos D, et al. The management of a simulated emergency: Better teamwork, better performance. Resuscitation 2011; 82: Irvine S and Martin J. Bridging the gap: from simulation to clinical practice. The Clinical Teacher 2014; 11:94-98 Blum RH, et al. Simulation-based assessment to identify critical gaps in safe anesthesia resident performance. Anesthesiology 2014; 120: Bould MD, et al. Assessment of procedural skills in anesthesia. Br J of Anaesth 2009; 103: McKinley RK, et al. Checklists for assessment and certification of clinical procedural skills omit essential competencies: a systematic review. Med Educ 2008; 42: Lammers RL, et al. Simulation-based Assessment of Paramedic Pediatric Resuscitation Skills. Prehospital Emergency Care 2009; 13: