CSCTR – Session 8 Dana Retová

 Rejects the standard view that amodal symbols represent knowledge in semantic memory  Cognition shares the same mechanisms with perception, action and introspection  Simulation ◦ A core form of computation in the brain ◦ Reenactment of perceptual, motor and introspective states acquired during experience ◦ As experience occurs, the brain captures the states across modalities and integrates them with a multimodal representation stored in memory

 Modal representation and imagery representing knowledge ◦ Epicurus, Kant, Reid  Behaviorists ◦ Imagery not sufficiently scientific  Cognitivists ◦ Amodal representation (feature lists, semantic networks, frames)  Elegant and powerful formalisms for representing knowledge  Could be implemented in AI

 No evidence supports the presence of amodal symbols in cognition  Grounding problem ◦ Traditional theories fail to explain how cognition interfaces with perception and action  Problem where the brain stores amodal symbols ◦ How is it consistent with neural principles of computation?

 Simulation  Situated action  Bodily states ◦ Modal representations are central to knowledge

 Cognitive Linguistics Theories ◦ Lakoff & Johnson (1980, 1999)  Abstract concepts are grounded metaphorically in embodied and situated knowledge  Theories of situated action ◦ Gibson (1979)  Role of environment in shaping cognitive mechanisms  Coupling of perception and action during goal achievement  Social interaction ◦ Research in robotics ◦ Dynamic systems as preferred architecture  Fixed representations do not exist

 Memory theories ◦ Glenberg (1997)  Memory is not just passive storage of information  Perception of relevant objects triggers affordances for action stored in memory  Reasoning about future actions relies on remembering affordances while suppressing perception of the environment  Social simulation theories ◦ How we represent the mental states of other people  We use simulations of our own minds  To feel someone else’s pain we simulate our own pain  Mirror neurons  Empathy, imitation, social coordination

 Perceptual Symbol Systems ◦ Synthetic approach  Implements standard symbolic functions  Type-token binding, inference, productivity, recursion, propositions ◦ A single multimodal representation system in the brain supports diverse forms of simulation across different cognitive processes  High-level perception  Working memory  long-term memory  conceptual knowledge  Convergence zone architecture (Damasio 1989, Simmons & Barsalou 2003)  Single representation system controlled by multiple simulation mechanisms

 ensemble of neurons within which many feedforward/feedback loops make contact.  It 1) receives forward projections from cortical regions located in the connectional level immediately below  2) Sends reciprocal backward projections to the originating cortices  3) Sends forward projections to cortical regions in the next connectional level; and  4) Receives projections from heterarchically placed cortices and from subcortical nuclei in thalamus, basal forebrain, and brainstem.

 Perceptual Inference  Perception-action coordination  Perception of space  Memory ◦ Implicit memory ◦ Explicit memory ◦ Working memory  Conceptual processing

 Vision and motion ◦ Goldstone (1995)  Association between shape and color ◦ Hansen et al. (2006)  Object’s natural color distort achromatic perception of the object toward the opponent color ◦ Motion (Freyd 1987, Shiffrar & Freyd 1990,1993)  Subjects simulate the visual trajectory beyond its actual trajectory  Also during apparent motion, simulation of possible action shapes the perception of motion ◦ Speech (Warren 1970) :  Lexical knowledge produces simulation in speech perception – missing phoneme simulation

 Simulations of potential actions ◦ Viewing an object grasped with a precision or power grip (grape vs. hammer) produces a simulation of the appropriate action (Tucker & Ellis 1998)  This is affected by  object’s orientation (Symes et al. 2007)  Size (Glover et al. 2004) ◦ Simulations of both grasping and functional actions (Bub et al 2007) ◦ Also name triggers simulation (Tucker & Ellis 2004) ◦ Hearing a word activates the articulatory action associated with producing it (Pulvermuller 2006) ◦ Perceived effort affects visual perception (Proffitt 2006)  Being tired from a run makes a hill look steeper  Carrying a heavy pack makes a path look longer

 Motor simulations ◦ Motor system constructs a feed-forward simulation of the action to guide and correct it (Grush 2004, Wolpert et al. 1999) ◦ Generating visual inferences about the anticipated actions of perceived agents (Wilson & Knoblich 2005)

 The perception of space is shaped by the body and it’s relation to the environment ◦ Locating objects has various difficulty along different axes  Vertical  easiest  Front-back  Left-right  Most difficult – bodily cues are lacking ◦ Perception of near space extends with arm length (Longo & Laurenco 2007)

 Results form simulation of perceptual memories  Repetition priming is strongest when the modalities of the memory and stimulus match (e.g. auditory) (Kirsner et al., 1989)  Repetition priming is strongest when perceptual details of the memory and stimulus match (e.g. orientation, size,…) (Jacoby&Hayman, 1987)  Imagining produces repetition priming similar to actual perception (Roediger&Blaxton, 1987)

 Multimodal simulations of previous episodes ◦ Important for constructing future events  The retrieval of a word stimulates the modal operations performed at encoding (Wheeler et al. 2000) ◦ Visual areas become active during retrieval following visual study while auditory areas become active following auditory study  Greater activation in modal areas when remembering something that really occurred than false memories (Slotnick & Schacter 2004)

 Absent stimulus is stored in working memory (Levy & GoldmanRakic 2000) ◦ To maintain working memory, neurons in the frontal lobes maintain a simulation of the absent stimulus in the modal system that processed it originally.  Some frontal regions maintain working memories of objects, other spatial locations, motion, textures, etc.  They are highly selective for the specific features  Visual imagery in working memory simulates visual processing (Finke 1989, Kosslyn 1980,…) ◦ Analogously, motor imagery, auditory imagery, etc.  Mental rotation of visual objects -> motor simulations of making them turn (Richter et al. 2000)

 Behavioral evidence ◦ When asked whether an property belongs to an objects subjects simulate properties to verify them (Solomon & Barsalou 2004)  Lesion evidence ◦ Lesions in one modality – losing categories that rely on it for processing (Damasio 1994, …)  E.g. damage to visual areas – losing of ability to categorize animals (visual processing is dominant)  Damage to motor areas – categorization of tools  Neuroimaging evidence (Martin 2001, 2007) ◦ When processing conceptual knowledge, brain areas representing properties are active

 Perceptual simulation  Motor simulation  Affective simulation

 Situation models ◦ Evidence of modal representations in language comprehension  Spatial representation (Bower & Morrow 1990)  People confused pictures with text (Intraub & Hoffman 1992)  Replacing words with pictures did not disrupt sentence processing (Potter 1986)

 Subjects read a sentence and then processed a picture that either matched or mismatched something implied by the sentence ◦ “The ranger saw the eagle in the sky” ◦ Picture of an eagle – wings outstretched or folded  Visual irrelevant information interferes with spatial inferences (Fincher-Keifer 2001)

 Verbs for head, arm and leg actions produce head, arm and leg simulation in the respective areas of the motor system (Pulvermuller 2005)  When action to make a response is consistent with text meaning, the response is quicker (Glenberg & Kaschak 2003)  Subjects simulate corresponding motion through space (Richardson et al. 2003)  Positive/negative valence (Meier & Robinson)  High/low power (Schubert 2005)

 Subjects’ faces configured according to sentences with emotional content (Havas 2007)  When facial emotion matched the content comprehension was better  Gesture ◦ Producing gestures helps speakers retrieve words whose meaning are related to the gestures (Krauss 1998) ◦ Also help listeners comprehend what speaker says ◦ Children can gesture before speaking

 Physical reasoning ◦ Gear, pulleys ◦ Driven by spatial simulation ◦ Sketchy, not holistic and detailed  Abstract reasoning ◦ Content effects ◦ Reasoning about time using space domain - metaphors

 Embodiment effects ◦ Activating elderly stereotype causes people to walk slowly and to perform lexical decision slowly (Dijksterhuis & Bargh 2001) ◦ Engaging the smiling musculature produces positive affect (Strack et al. 1988)  Social mirroring ◦ Individual differences in the ability to simulate other people’s mental states correlate with rated empathy (Jackson et al. 2005)  Development ◦ Mirroring, object permanence…