A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok

Mixed Reality for Training Training + Mixed Reality(MR) = Improved Training –i.e Military [x], industrial[x], medical [x] –MR is used during the training session After Action Review (AAR) –Playback and review of one’s training session i.e. AAR in the HPS –Currently: video reviews –helpful for bringing trainees to competency AAR + MR = ?

After Action Review Students need: –Repetition, directed instruction, Feedback Educators need –Assessment Tools Video Based Review offers: –Assessment Tools and Repetition: video playback (FF,REW,etc) –instruction and feedback: Expert must be present during student review

Video Based AAR Widely used Limitations –Fixed, exocentric viewpoint –Minimal interaction (Play, Pause, etc) –See only what the camera sees (i.e occlusions are problematic) –Lost information

MR Based AAR Video review = lost information –Goal: maximize information flow with MR Advantages of MR systems in general: –Egocentric viewpoint –interactivity – i.e. TUIs –Collocated virtual information (i.e. gaze) Can we enhance AAR with MR?

Innovations The Augmented Anesthesia Machine Visualization and Interactive Debriefing System (AAMVID) –MR based AAR system for Anesthesia education –Educator version: AAMVID-E –Student version: AAMVID-S User evaluation (19 students, 3 educators) General goal: Evaluate MR for use in AAR

Previous Work AAR [x],[x],[x] Recording Virtual Experiences –[x],[x],[x] AAR of Virtual Human Experiences –IPSViz[x]

Anesthesia Education and MR The Augmented Anesthesia Machine [x] –merges and abstract simulation of an anesthesia machine with a real machine (magic lens + TUI) –Improves learning

AAR in Anesthesia Education Currently Video Based –Students perform a test (i.e. a fault test) –Educators and students review and critique performance

AAR in Anesthesia Education MR system: AAMVID –Based on the AAM –Students (AAMVID-S): View their own or an expert’s interaction in situ Follow a directed tutorial guided by the expert’s interaction visualization –Educators (AAMVID-E): Can visualize and filter aggregate student data AAMVID addresses video review limitations

Student AAMVID-S System playback controls Gaze visualization interaction visualization

Student AAMVID-S System 3 Visualization Modes –User Review View what the student did –Expert Review View what the expert did –Expert Tutorial Collocates: –expert’s gaze and interaction event boxes –Real time abstract simulation of the student’s machine Interaction with the real machine while mimicking expert Hands-on experience in correcting the fault

AAMVID-S Evaluation Usability and performance evaluation of AAMVID-S 19 psychology students: Day 1 (90 min) Training with AAM Day 2 (90 min) 1.Testing 2. AAR of Tests with AAMVID

Day 2: Testing Three Fault Tests –Experimenter caused a problem in the machine i.e. disabled a the flow of nitrous oxide –Participant had to decide Whether or not a fault was present What the fault was (if present) How to correct the fault (if present)

Metrics Repeated Measures design: Performance Measures: –Participants were asked questions (each scored 0-4) Fault absent or present What was the fault? How does the fault affect the patient? How to correct the fault? Confidence Measures: –Self reported confidence ratings for above questions (0- 4)

Results Insert graphs that show significant improvements in performance and confidence.

Discussion Main results: –AAMVID Improved machine fault understanding Performance question answers improved. –AAMVID increased confidence in fault correction ability Confidence scores increased –Users prefer Expert Tutorial Mode

Educator AAMVID-E System Aggregate Gaze Heat Mapping Interactive Filtering (i.e. on spatial cog data)

Educator AAMVID-E System Interactive Filtering (i.e. on spatial cog data) Interaction Graph (markov model of class interaction)

Informal Evaluation 3 Experts in anesthesia education –Dr. Samsun Lampotang –David Lizdas –Dr. Nicholas Gravenstein

Observations Magic lens vs desktop? What worked and what didn’t work

Conclusions Presented AAMVID –Improved student understanding and confidence –Offered needed assessment abilities to educators that were not available before MR based AAR – can offer visualizations and interactivity that video based AAR cannot