The Embodiment of Mind as a Natural Result of Interactive Dynamics Michael J. Spivey Department of Cognitive Science University of California, Merced 16th International Summer School in Cognitive Science, New Bulgarian University, Sofia, Bulgaria, 2009

Day 1: Embodied cognition is unavoidable in an organism that allows its subsystems to richly interact Day 2: Findings in the Embodied Cognition literature Day 3: How classical symbolic cognitive science would like to interpret embodied cognition findings Day 5: Computational models that exploit the embodiment of cognition Day 4: Findings of the motor system directly constraining cognitive processes COURSE OVERVIEW

The Embodiment of Mind Temporal Dynamics in Neuronal Population Coding Temporal Dynamics in Spoken Word Recognition Temporal Dynamics in Sentence Processing Mental Trajectories in Neuronal State Space Temporal Dynamics in Question Answering OUTLINE

The Embodiment of Mind Temporal Dynamics in Neuronal Population Coding Temporal Dynamics in Spoken Word Recognition Temporal Dynamics in Sentence Processing Mental Trajectories in Neuronal State Space Temporal Dynamics in Question Answering OUTLINE

Embodiment of Language (e.g., Lakoff & Johnson, 1980, 1999; Sweetser, 1998; Gibbs, 2006; Langacker, 2008; Talmy, 2006) Embodiment of Cognition (e.g., Barsalou, 1999; MacWhinney, 1999; Glenberg, 1997; Spivey, 2007; Varela, Thompson, & Rosch, 1992) Embodiment of Perception (e.g., Gibson, 1979; Turvey, 1992; Hommel et al., 2001) Embodiment of Cognitive Development (e.g., J. Mandler, 1992; L. Smith, 2005) Embodiment of Artificial Intelligence (e.g., Brooks, 1991; Ballard et al., 1997; Roy, 2005; Steels, 2003)

Page 1 of 2 Motor areas are active when hearing action verbs (Pulvermüller, 1999) Sentences evoke perceptual simulations (Stanfield & Zwaan, 2001) Sentences evoke motor simulations (Glenberg & Kaschak, 2002) Real and fictive motion priming influences reasoning about language (Boroditsky & Ramscar, 2002; Matlock, Ramscar, & Boroditsky, 2005) Image schemas activated by language interfere with perceptual input (Richardson, Spivey, Barsalou, & McRae, 2003; Bergen et al., 2007) Summaries of much of this work can be found in Pecher & Zwaan (2005).

Page 2 of 2 Sync in postural sway emerges during mutual conversation (Shockley, Santana, & Fowler, 2003) Sync in eye movements at shared scene emerges during conversation (Richardson & Dale, 2005) Fictive motion sentences induce more directional scanning of scene (Richardson & Matlock, 2007) Motion words influence visual perception of motion, and vice versa (Meteyard, Bahrami, & Vigliocco, 2007; Meteyard et al. 2008) Direction words heard during a reach cause curvature in the trajectory (Boulenger et al., 2006; Nazir et al., 2008)

The Embodiment of Language Temporal Dynamics in Neuronal Population Coding Temporal Dynamics in Spoken Word Recognition Temporal Dynamics in Sentence Processing Mental Trajectories in Neuronal State Space Temporal Dynamics in Question Answering OUTLINE

(Tyler & Spivey, 2005)

Wernicke’s Area

candle candy

Population Code for “candy”

Population Code for “candle”

0 ms

100 ms

200 ms

300 ms

400 ms

Population code for recognizing “candy”

Population code for recognizing “candle”

0ms

100ms

200ms

300ms

Population code for recognizing “candle” 400ms

frontal 3/4 profile profile 1/4 profile back-of-head After Stimulus Onset (ms) Cumulative Response (spikes) Perrett, Oram, & Ashbridge (1998)

Temporal dynamics of population coding (Rolls & Tovee, 1995) Cumulative Information (bits) (ms) If normal eye movements occur 3-4 times per second, then a population code rarely gets enough time to asymptote!

Multi-dimensional scaling of the 14 cells (Rolls & Tovee, 1995)

Trajectories in state space for recognizing faces and objects

Trajectories in state space as a result of free-viewing of a scene

McClelland & Elman’s (1986) TRACE Model of Speech Processing

(McClelland & Elman, 1986) Normalized Activation TRACE Model of Speech Perception 1-sum candle candy pencil penny

Energy Landscape Attractor Basins

Vector Field Attractor Basins

Vector Field Attractor Basins

Vector Field Attractor Basins

On the equivalence between dynamic neural patterns and a trajectory through mental state space (Onnis & Spivey, submitted)

Fodor’s (1983) modularity of mind has been giving way to a continuity of mind. Visual Perception (Motter, 1993; Sekuler, Sekuler, & Lau, 1997; Spivey & Spirn, 2000) Spoken Word Recognition (Elman & McClelland, 1988; Allopenna et al., 1998) Sentence Processing (MacDonald & Seidenberg, 1993; Tanenhaus & Trueswell, 1995 Chambers, Tanenhaus, & Magnuson, 2004) Conceptual Integration and Blending (Fauconnier & Turner, 2002; Coulson, 2001) Construction Grammar, eschewing the distinction between core grammar and periphery (Fillmore, 1988; Goldberg, 2003)

The Embodiment of Language Temporal Dynamics in Neuronal Population Coding Temporal Dynamics in Spoken Word Recognition Temporal Dynamics in Sentence Processing Mental Trajectories in Neuronal State Space Temporal Dynamics in Question Answering OUTLINE

Headband-mounted Eyetracking

Why headband-mounted eyetracking? Saccades are frequent: 3-4 times per second. Saccades are promiscuous (low threshold for execution), and thus sensitive to subtle probabilistic biases. Eye position provides a continuous measure without interrupting processing. Eye movements are largely resistant to strategic control. Headband allows ecologically valid continuous interaction between human and environment

Vector Field thresholds for a saccade

(Tanenhaus, Spivey-Knowlton, Eberhard, & Sedivy, 1996) “Pick up the candle.”

(Tanenhaus, Spivey-Knowlton, Eberhard, & Sedivy, 1996) “Pick up the candle.”

Probability of Fixation (ms) (Spivey-Knowlton, 1996)

McClelland & Elman’s (1986) TRACE Model of Speech Processing

(Allopenna, Magnuson & Tanenhaus, 1998)

Eye-movement data Probability of Fixation (ms) (Spivey-Knowlton, 1996)

(Reali, Spivey, Tyler, & Terranova, 2006) “Pick up the…” Saliency Map

(Reali, Spivey, Tyler, & Terranova, 2006) “Pick up the ca…” Saliency Map

(Reali, Spivey, Tyler, & Terranova, 2006) “Pick up the cand…” Saliency Map

(Reali, Spivey, Tyler, & Terranova, 2006) “Pick up the candle.” Saliency Map

Continuous Non-ballistic Movements (Spivey, Grosjean, & Knoblich, 2005)

Mouse-click start box at bottom (Spivey, Grosjean, & Knoblich, 2005)

“Click the ladle” (Spivey, Grosjean, & Knoblich, 2005)

y x (Spivey, Grosjean, & Knoblich, 2005)

“Click the beetle” (Spivey, Grosjean, & Knoblich, 2005)

Graded spatial attraction toward phonological competitors visible in averaged trajectories (Spivey, Grosjean, & Knoblich, 2005)

The Embodiment of Language Temporal Dynamics in Neuronal Population Coding Temporal Dynamics in Spoken Word Recognition Temporal Dynamics in Sentence Processing Mental Trajectories in Neuronal State Space Temporal Dynamics in Question Answering OUTLINE

Incrementality in Syntax Put the apple on... Verb Noun Phrase Prep. Phrase Preposition Verb Phrase Sentence... Syntactic Ambiguity

In a Visual Context One-Referent Context Two-Referent Context (Tanenhaus, Spivey-Knowlton, Eberhard & Sedivy, 1995) “Put the apple on the towel in the box” “Put the apple that’s on the towel in the box” Ambiguous: Unambiguous:

One-Referent Context Tanenhaus, Spivey-Knowlton, Eberhard & Sedivy, 1995)

Two-Referent Context Tanenhaus, Spivey-Knowlton, Eberhard & Sedivy, 1995)

In a Visual Context ((Tanenhaus, Spivey-Knowlton, Eberhard & Sedivy, 1995)

Continuous Non-ballistic Reaching Movements

One-Referent Context Two-Referent Context “Put the apple on the towel in the box” “Put the apple that’s on the towel in the box” Ambiguous: Unambiguous: (Farmer, Anderson, & Spivey, 2007)

One-Referent Context, Ambiguous Sentence (Farmer, Anderson, & Spivey, 2007)

The Embodiment of Language Temporal Dynamics in Neuronal Population Coding Temporal Dynamics in Spoken Word Recognition Temporal Dynamics in Sentence Processing Mental Trajectories in Neuronal State Space Temporal Dynamics in Question Answering OUTLINE

YES NO (McKinstry, Dale, & Spivey, 2008) start “Is a thousand more than a million?”

YES NO (McKinstry, Dale, & Spivey, 2008) start “Should you brush your teeth every day?”

YES NO (McKinstry, Dale, & Spivey, 2008) start “Is murder sometimes justifiable?”

(McKinstry, Dale, & Spivey, 2008)

The Embodiment of Language Temporal Dynamics in Neuronal Population Coding Temporal Dynamics in Spoken Word Recognition Temporal Dynamics in Sentence Processing Mental Trajectories in Neuronal State Space Temporal Dynamics in Question Answering OUTLINE

Toward an equivalence between dynamic neural patterns and a trajectory through mental state space (Onnis & Spivey, submitted)

“The new form of insight can perhaps best be called Undivided Wholenesss in Flowing Movement. This view implies that flow is, in some sense, prior to that of the ‘things’ that can be seen to form and dissolve in the flow.” -David Bohm, Wholeness and the Implicate Order (1980)

Slides for questions:

A. Weights over time for alternative reaches B. Simulated reach movements Time steps X (cm) Y (cm) Relative Activation “candy” control condition cohort condition “candy” “candle” foil “book” foil reach to foil reach to target control cohort * (Spivey, Dale, Grosjean, & Knoblich, in press)

Quasilinguistic thought while learning racquetball

Multi-dimensional scaling of the 14 cells (Rolls & Tovee, 1995)

Temporal dynamics of population coding (Rolls & Tovee, 1995)

Trajectories in state space for recognizing faces and objects

Trajectories in state space as a result of free-viewing of a scene

A B Q P R X Y m n Symbolic Dynamics