1. Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014

2. Assessment Formative Summative

3. Formative Assessment Assessment for learning

4. Summative Assessment Assessment of learning

5. Evaluation Methodology Bould MD, et al. Assessment of procedural skills in anesthesia. Br J of Anaesth 2009; 103:472-483

6. Methodology: Global Rating Scale Overall performance by categories Scores (Likert) Drawbacks

7. Global Rating Scale Blum RH, et al. Simulation-based assessment to identify critical gaps in safe anesthesia resident performance. Anesthesiology 2014; 120:129-41

8. Methodology: Checklists Delineated actions Weighted, timed, cumulative... Adverse events Drawbacks

9. Checklists Schwid, et al. Evaluation of anesthesia residents using mannequin-based simulation, a multi institutional study. Anesthesiology 2002; 97:1434-1444

10. Methodology: “Time To” Timing of critical actions Pace Motion analysis

11. Methodology: Survey Data Self assessments Psychometrics: Pre and Post Testing

12. Survey Data Lammers RL, et al. Simulation-based Assessment of Paramedic Pediatric Resuscitation Skills. Prehospital Emergency Care 2009; 13:345-356

13. Methodology: Biometrics “Technometrics?” Mannequin physiology

14. Validity The Assessment Method The Simulation

15. Validity: “Face” General acceptability of the tool used for the task required Does this make sense?

16. Validity: “Content” Does the content match what’s to be assessed? Task Analysis The Experts

17. The Delphi Method BOGSAAT

18. Validity: “Construct” Does this reflect true performance? Experts Role of experience Surveys

19. Validity: “Construct” Statistical Analysis: Discrimination: “D Statistic” ANOVA “Concurrence”

20. Tool Validation Reid J, et al. The Simulation Team Assessment Tool (STAT): Development, reliability and validation. Resuscitation 2012; 83:879-886

21. Validity: “Criterion” Or “Extrapolation” Similar results to other performance measures “Predictive”

22. Consistency: Internal Requires realism Reflects test quality Pilot Testing

23. Consistency: Statistical Analysis Do testers do well or poorly on the same items? Generalizability: “G statistic”

24. Reliability: Inter-Participant Set-up Simulator The Subject

25. Reliability: The Rater Inter-rater reliability (external reliability) Statistics: IRR, ICC

26. Reliability: The Rater Rater training Rater refreshing Scoring

27. Reliability: The Observation Real time Recording Bias

28. Validated Tools Cheng A, et al. Designing and Conducting Simulation-Based Research. Pediatrics 2014; 133:1090-1101.

29. Take Away Rigor in realism, rehearsing and piloting Mix Checklists and GRS Supplement with survey, psychometric, device data, etc More tasks with more (recorded) views

30. Further Sourcing Schwid, et al. Evaluation of anesthesia residents using mannequin-based simulation, a multi institutional study. Anesthesiology 2002; 97:1434-1444 Devitt, et al. The validity of performance assessment using simulation. Anesthesiology 2001; 95:36-42 Devitt, et al. Testing internal consistency and construct validity during evaluation of performance in a patient simulator. Anesth Analg 1998; 86:1160-1164 Devitt JH, Kurrek MM, Cohen MM, Fish K, Fish P, Murphy PM, Szalai JP: Testing the raters: Inter-rater reliability of standardized anaesthesia simulator performance. Can J Anaesth 1997; 44:924–8 Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R: Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998; 89:8-18 Boulet JR, et al. summative assessment in medicine: the promise of simulation for high-stakes evaluation. Acad Emerg Med 2008; 15:1017-1024 Cheng A, et al. Designing and Conducting Simulation-Based Research. Pediatrics 2014; 133:1090-1101. Furman GE et al. Quality Assurance Best Practices for Simulation-Based Examinations. Qual Assurance Best Practices 2014; 5:226-231 Reid J, et al. The Simulation Team Assessment Tool (STAT): Development, reliability and validation. Resuscitation 2012; 83:879-886 Siassakos D, et al. The management of a simulated emergency: Better teamwork, better performance. Resuscitation 2011; 82:203-206 Irvine S and Martin J. Bridging the gap: from simulation to clinical practice. The Clinical Teacher 2014; 11:94-98 Blum RH, et al. Simulation-based assessment to identify critical gaps in safe anesthesia resident performance. Anesthesiology 2014; 120:129-41 Bould MD, et al. Assessment of procedural skills in anesthesia. Br J of Anaesth 2009; 103:472-483 McKinley RK, et al. Checklists for assessment and certification of clinical procedural skills omit essential competencies: a systematic review. Med Educ 2008; 42:338-349 Lammers RL, et al. Simulation-based Assessment of Paramedic Pediatric Resuscitation Skills. Prehospital Emergency Care 2009; 13:345-356