1. NMC Online Conference December 2005 Assessing Learning in a MORPG Patricia Youngblood, PhD Director of Evaluation, SUMMIT (Stanford University Medical Media & Information Technology) Phillip M. Harter, MD Residency Program Director Division of Emergency Medicine Stanford University School of Medicine

2. NMC Online Conference December 2005 Comparison of the Virtual ED & HPS Human Patient Simulator Virtual ED (Emergency Department)

3. NMC Online Conference December 2005 Virtual ED (Emergency Department)

4. NMC Online Conference December 2005 Debriefing with instructor Student being assessed Standardized team member

5. NMC Online Conference December 2005 Study Design: Pretest/Posttest 30 medical students & interns Virtual ED Simulator (n=16) pretest case  4 training cases  posttest case Human Patient Simulator “HPS” (n=14) pretest case  4 training cases  posttest case Is there a significant difference in performance between the two student groups?

6. NMC Online Conference December 2005 EMCRM Rating Scale of Team Performance (1-5) Knowledge of the environment Anticipation of & planning for potential problems Assumption of leadership role Communication with other team members Distribution of workload/delegation of responsibility Attention allocation Utilization of information Utilization of resources Recognition of limitations/call for help early enough Professional behavior/interpersonal skills Overall team leadership skills (1= not acceptable; 5=excellent)

7. NMC Online Conference December 2005 4. Communication with other Team Members 12345 Average performance (3): Communicates clearly with all team members; speaks clearly, calls team members by name, can be heard easily Below Average (1-2): Cannot be heard easily, doesn’t identify to whom he is speaking, talks ‘down’ to other members, does not close the communication loop Above Average (4-5): Closes the communication loop, thinks/talks out loud so other members can follow his problem solving approach Guidelines for Scoring Team Leadership in the Virtual ED

8. Comparison of Pre & Post Mean Scores Mean Scores Pretest Sum ScoresPosttest Sum Scores *Significant at p=0.000 0.00 10.00 20.00 30.00 40.00 50.00 HPS Group Pretest Sum ScoresPosttest Sum Scores Pretest Sum Scores Posttest Sum Scores Virtual ED Group

9. NMC Online Conference December 2005 Results Summary Users’ performance rating scores improved between pretest case and posttest case with both the MORPG and the Human Patient Simulator; Analysis of performance data shows no statistically significant differences between the two treatment groups; Reliability of the EMCRM rating scale was high (Cronbach’s alpha =.96); Users liked both types of simulation training.

10. NMC Online Conference December 2005 The Stanford Research Team