1. The Science of Learning and the Virtual Anesthesia Machine: Benefits of "schematic" simulations in learning about complex systems Ira Fischler Simulation Faculty Learning Community May 2008

2. Colaborators: Sem Lampotang (Anesthesiology) Cynthia Kaschub (Psychology) David Lizdas (Anesthesiology) And for jump-starting this effort: Sue Legg (Director, Partnership for Global Learning)

3. Plan for the talk: “Learning with understanding:” The idea of mental models in psychology and education How multimedia presentations can boost learning Potential advantages of simulation Transparent simulations and understanding The Virtual Anesthesia Machine (VAM) Learning with Transparent versus Opaque VAM Bridging abstract and concrete models: Mixed Reality and the Augmented Anesthesia Machine A little bit about individual differences

4. The mini-science of learning What makes a difference?  Amount of practice (and the Power Law)  Distribution of practice (and the Spacing Effect)  Quality of practice (and Depth of Processing) Making the information distinctive Building appropriate “mental models” “I’m doing great in all my other classes. I read the book, came to class, outlined the material, and made flash cards, and still got a C.” “Well, did you understand the material?” “I thought I did…”

5. Mental models and schemas in comprehension “If the balloons popped, the sound wouldn’t be able to carry since everything would be too far away from the correct floor. A closed window would also prevent the sound from carrying, since most buildings tend to be well insulated. Since the whole operation depends on the steady flow of electricity, a break in the middle of the wire would also cause problems. Of course, the fellow could shout, but the human voice is not loud enough for the sound to carry that far. An additional problem is that the string could break on the instrument. Then there would be no accompaniment to the message. It is clear that the best situation would involve less distance….

7. Mental models in cognitive science Term first used by Kenneth Craik (’43) “If the organism carries a “small-scale model” of external reality and of its own possible actions within its head, it is able to try out various alternatives, conclude which is the best of them, react to future situations before they arise, utilise the knowledge of past events in dealing with the present and future, and in every way to react in a much fuller, safer, and more competent manner to emergencies which face it.” (Craik, The Nature of Explanation, 1943) Quality of the model depends on how well it captures the features of the domain that are critical for the task at hand

8. Understanding problems (Greeno, 1977) Our internal representation (or model) of the problem should have  accurate CORRESPONDENCE between relevant elements in the world and model  good COHERENCE between elements in the model  appropriate links to PRIOR KNOWLEDGE that can aid problem solving environment mental model of problem correspondence coherence links to prior knowledge

9. Little things (can) mean a lot (aka the devil’s in the details) Subtle changes in problem “framing” can have drastic effects on performance  Effects of analogy on solving the “X-ray” problem Preceded by bulb filament problem  “fragile glass” framing:33% then solve X-ray  “laser intensity” framing: 69% then solve X-ray  Effects of lives lost/saved on risky decisions: Disease control programs, one more risky  “Lives saved” framing: 22% choose risky action  “Lives lost” framing: 75% choose risky action

10. Came before text, historically Illustrations and drawings  To illuminate structure, function and relations Animations and videos  To make system dynamics visible Interactive simulations  To actively explore cause-effect dynamics, test hypotheses, etc. Advantages of “multimedia” can be dramatic: Pictorial Representations

11. Mayer’s work on Multimedia (e.g., How lightning forms) Compares..  text-only to text-with-illustration (often schematic)  Narration-only with narration-plus-animation Tests..  Retention by free recall of presented facts  Transfer (understanding?) by generating solutions to: Redesign (“how could you decrease lightning intensity?”) Troubleshooting (“how could there be clouds, but no lightning?”) Prediction (“what would happen with lower air temperature?”) Abstraction of Principles (“What causes lightning?”)

12. Retention and transfer with MM Retention: Modest MM gains  Across 6 studies, 23% gain, 0.67 effect size Transfer: Dramatic MM gains  Across 6 studies, 89% gain, 1.50 effect size

13. Potential advantages of Computer-based simulation Cost: cheap systems, easy to replace, low risk Track performance and provide “just-in-time” feedback on performance “Virtually Real” when needed But Reality can be played with:  Increase likelihood of rare but important events  Increase salience of important features  Present “hyper-real” depictions of space and time  Make the abstract concrete, and the invisible visible

14. Instructional Choice-Points What do we want them to learn?  Declarative knowledge, procedural skills  Immediate or long-term retention  Reproductive or creative, flexible learning How do we structure the learning?  Amount of “grounding” in the domain  Balance of guided (reception) and free (discovery) learning  Amount of online assessment and intelligent tutoring  Student-tailored, or one-size-fits-all

15. Opaque versus Transparent Reality Opaque representation: Simulation may be closely analogous to the physical system (iconic, concrete, high-fidelity, virtual reality) but hides underlying structure, functions and relations Transparent representation: Simulation sacrifices physical fidelity but makes underlying aspects of system overt (abstract, idealized, schematic, symbolic)

16. Transparency in simulations Hollan’s STEAMER (1981) Goldstone’s Concreteness Fading (2004) Butcher’s simplified diagrams (2006) Debate focusses on “extent of fidelity” and whether detail helps or hurts Little direct comparison of simulation formats

17. The Opaque-Reality VAM

18. The Transparent-Reality VAMVAM

19. The Virtual Anesthesia Machine wide use, little data 10 man-years of development time Available for free to individuals on the web Over 10,000 registered users Many positive reviews, both formal and informal Our goal: assess the effectiveness of VAM’s Transparent Reality approach to simulation

20. Training Session 30-page instructional guide developed  Provides foundation of knowledge About anesthesia About the anesthesia machine and its subsystems Guided tour of several subsystems  Breathing circuit  Mechanical ventilation  Manual ventilation Stresses visualization of dynamics using VAM

21. Workbook: Sample text Question 1: elimination of CO 2. Are the gases exhaled by a patient “scrubbed” of CO 2 before entering the bellows during mechanical ventilation? Demonstration using VAM Simulation: ____ Click “Reset” to start simulation afresh ____ Point to the O 2 flowmeter control knob to enlarge it, then click-and-hold, and drag it counterclockwise until the O 2 bobbin inside is about halfway up the tube.

22. Workbook: sample text (cont’d) What does this do? What happens to the flow of O 2 from the supply line?  Opening the valve increases the flow of O 2 from the supply line into the breathing circuit. Trace along its route through the plumbing. Where does it wind up?  It depends. For example, If mechanical ventilation is selected, but not on, the O 2 flows “backward” through the CO 2 absorber, past the bellows and into the scavenger system

23. Performance on Day 2 Tests: undergraduate health majors

24. Performance on Day 2 Tests: 2 nd -year medical students

25. Performance on AAA Board Exam Review questions (4AFC)

26. Judgments about VAM Confidence Judgments  On Component function: Significantly higher for Transparent VAM (p <.01)  On System dynamics: Marginally higher for Transparent VAM (p <.15) Preferences for additional study  17 of 20 in Transparent group (UG) prefer TR VAM  11 of 20 in Opaque group prefer (UG) TR VAM  2 in TR, 7 in OR, think both would be preferable to either  Similar trends among medical students; more want both

27. Where to next? Combination and order effects  Goldstone’s “concreteness fading” method? More precise tests of transfer  Transfer to procedural skill: does TR improve error detection and response? Hybrid simulations: John Quarles’ project

28. The Augmented Anesthesia Machine (AAM) Integrating transparent and realistic representations with “mixed-reality” simulation John Quarles and his Magic Lens

29. Declarative and Procedural Knowledge with VAM and AAM Two groups of undergrads Training:  Introduction to AM with VAM  “positioning” components within the actual AM  Five step-through exercises with VAM or AAM Day 2 Testing  Declarative: Board Exam Questions  Procedural: Find a machine fault in the AM

30. Performance with VAM vs. AAM

31. Abstract and Concrete Knowledge “Although the VAM may offer improved abstract knowledge, participants found it difficult to transfer this knowledge to the concrete anesthesia machine. This is precisely the concern that anesthesia educators have had with the VAM. For example, many VAM participants understood the abstract concept of the inhalation valve and they correctly answered the written questions regarding the gas flow in the valve. However, during the fault test, they could not perform the mental mapping between the abstract representation of the VAM inhalation valve and the concrete representation of the real anesthesia machine inhalation valve. Thus, it was difficult for VAM participants to apply their abstract knowledge to a concrete problem, such as the problem presented in the fault test.”

32. Role of Spatial Abilities? Three tests of spatial cognition  Arrow-pointing working memory (small-scale)  Perspective-taking (mid-scale)  Navigating virtual environment (large-scale) Correlations of spatial abilities and performance tend to be larger with VAM than AAM  Suggests those with strong visualization skills can compensate for impoverished materials

33. What we’ve learned Dynamic simulations can improve comprehension of, and memory for, complex systems, BUT - Different kinds of simulation are optimum for different kinds of learning  So we need to know the goal of training Experience with both abstract (schematic) and concrete (hi-fidelity) simulations may be optimum  So we may need an integrated approach Individual differences in domain-specific skills and abilities will impact effectiveness of representations  But we need to know how much

34. It takes a Village (or at least a Learning Community) Cognitive/human factors psychologists Usability analysts Instructional psychologists and educators Simulation designers and engineers Domain experts and professionals

35. Thanks to all those RA’s Emily McAlister Jonathan Greenwood Julianna Peters Shannon Bowie Sheila Holland Trudy Salmon