Imaginary Worlds and Grounded (Embodied) Cognition John B. Black Teachers College Columbia University

Research with Gordon Bower Studied Students Reading and Remembering Stories Found That Switches in Point of View Lead to Longer Reading Times and Memory Errors – Spatial Layout Found That Actions That Changed The Story World Were Remembered Better Seemed to be Story World Separate From Symbolic Content of Story (Mental Model)

Symbolic Content vs Model World Johnson-Laird Proposed Mental Models That Involved Imagining Models of Logical Expressions Kintsch Proposed Situation Models of Textual Content Separate from Textbase of Propositional Networks Philosophers like Frege and Carnap Distinguish Between Sense and Reference

My Research at TC Shifted from Text to Multimedia What is the Nature of These Mental Referent or Imaginary Worlds? How Does One Teach For Imaginary World Level of Understanding?

World of a Cathedral Multimedia System to Learn About St. John’s Cathedral (Van Esselstyn and Black) Learned a lot from just Floor Plan (survey map knowledge) and Text Providing Virtual Tour (dynamic route map knowledge) only helped when given rationale for spatial layouts Constructing this Cathedral World needed reasons for spatial layout

Multimedia Cathedral

Research with Maggie Chan Crucial Element of Imaginary World Reasoning is Functional Relations Functional Relations Describe How One Entity Changes as a Function of Change in Another Deeper Lever Understanding Than Structural Causal Relations Showed Direct Manipulation Animation Good Way to Learn This

Roller Coaster Simulation Game

Roller Coaster Game Studies Snapshots (like Comics) Better than System Controlled Animation Consistent with Many Studies of Animation in Tversky Survey “Direct Manipulation” of one Factor with Animation of Other Factors Gave Better Learning (Chan and Black)

Experimental Design IV1: Presentation Formats: text-only, text-plus-static diagrams, text-plus-direct-manipulation animation IV2: System Complexity: simple (Energy conversion in a playground swing ), moderate (Energy conversion in a roller coaster ride), complicated (Energy conversion of pole vaulting) Dependent Measures: DV1: Verbal Recall (summary) DV2: Visual Recall (drawing) DV3: What-if Scenarios DV4: What’s wrong here? Case study DV5: Transfer

Direct Manipulation Animation Simple system --- a playground swing

Direct Manipulation Animation Complex system --- pole-vaulting

7 th Grade What-if scenarios What-if scenarios

6 th Grade

Result Varies within Type of Student and System Complexity General form seems to be for simple systems and capable students text is sufficient Also for capable students and intermediate systems text plus diagrams sufficient But for complex systems for any students and less capable students for any system need direct manipulation animation

Video Games for Learning There is a Lot of Hype About Video Games as Learning Environments However, Almost No Studies That Show This – in fact, Some Show Reverse But We Have Found Video Games Good as Part of Learning Environment But Not By Themselves

Video Games as Preparation for Future Learning Civilization

Preparation for Future Learning Video Game Study Compared Experts in Civilization Formally Learning New Material Civilization Experts Were Better at Formally Learning New Material – Particularly Systems Related Particularly Systems Related Thus Video Games May Provide Experiences as Experiences as Preparation for Future Learning

Reflective Questions with Games Studied Business Student Learning From Business Simulation Game (from HBS) Students Learned Much More If Game Supplemented with Reflective Questions About Strategies Used and Relation to Background Reading Army Has Found Learning With America’s Army Game Occurs in After Action Reports Which Provide Reflection

Grounded (Embodied) Cognition Currently Think Imaginary Worlds Involves Grounded or Embodied Cognition This is a Perceptual Simulation to Supplement (Provide Mental Referents) Recent Neural Imaging Studies Show Perceptual Areas of Brain Activated in Symbolic Tasks in Addition to Symbolic Areas

How To Investigate This? Glenberg Text Comprehension ARIAREAL???

ARIA Project – Associating Real and Imaginary Agents ARIA Project Provides Embodiment by Having Students Work With Robot Simulations on Computer Screen Then Construct Robots to Do It In Real World Also Provides Embodiment By Having Students Act Out Using There Own Bodies What they Want Robots to Do Being Done as After School Program at The School at Columbia U.

REAL—Reflective Agent Learning environment Provides Embodiment by Allowing Learner to Input Knowledge Representations in Head of Agents Then See How Agent with Knowledge Behaves in Virtual World Various Depictions of Results Reflects What is Good and Bad About Knowledge Then Try Again

REAL Business

REAL Business (Design Mode)

REAL Business (Game/Reflection Mode)

Constructing Dynamic Imaginary Worlds Virtual Worlds with Animated Agents can be Effective for Learning if Spatially Meaningful Causally Transparent Directly Manipulate Factors and See Animated Results Gets Internalized as Imaginary World Augmented Reality Promises to Give Same Benefits Better Integrated with Real World