1. The Embodiment of Mind as a Natural Result of Interactive Dynamics Michael J. Spivey Department of Cognitive Science University of California, Merced 16th International Summer School in Cognitive Science, New Bulgarian University, Sofia, Bulgaria, 2009

2. Day 1: Embodied cognition is unavoidable in an organism that allows its subsystems to richly interact Day 2: Findings in the Embodied Cognition literature Day 3: How classical symbolic cognitive science would like to interpret embodied cognition findings Day 5: Computational models that exploit the embodiment of cognition Day 4: Findings of the motor system directly constraining cognitive processes COURSE OVERVIEW

3. The Embodiment of Mind Temporal Dynamics in Neuronal Population Coding Temporal Dynamics in Spoken Word Recognition Temporal Dynamics in Sentence Processing Mental Trajectories in Neuronal State Space Temporal Dynamics in Question Answering OUTLINE

4. The Embodiment of Mind Temporal Dynamics in Neuronal Population Coding Temporal Dynamics in Spoken Word Recognition Temporal Dynamics in Sentence Processing Mental Trajectories in Neuronal State Space Temporal Dynamics in Question Answering OUTLINE

5. Embodiment of Language (e.g., Lakoff & Johnson, 1980, 1999; Sweetser, 1998; Gibbs, 2006; Langacker, 2008; Talmy, 2006) Embodiment of Cognition (e.g., Barsalou, 1999; MacWhinney, 1999; Glenberg, 1997; Spivey, 2007; Varela, Thompson, & Rosch, 1992) Embodiment of Perception (e.g., Gibson, 1979; Turvey, 1992; Hommel et al., 2001) Embodiment of Cognitive Development (e.g., J. Mandler, 1992; L. Smith, 2005) Embodiment of Artificial Intelligence (e.g., Brooks, 1991; Ballard et al., 1997; Roy, 2005; Steels, 2003)

6. Page 1 of 2 Motor areas are active when hearing action verbs (Pulvermüller, 1999) Sentences evoke perceptual simulations (Stanfield & Zwaan, 2001) Sentences evoke motor simulations (Glenberg & Kaschak, 2002) Real and fictive motion priming influences reasoning about language (Boroditsky & Ramscar, 2002; Matlock, Ramscar, & Boroditsky, 2005) Image schemas activated by language interfere with perceptual input (Richardson, Spivey, Barsalou, & McRae, 2003; Bergen et al., 2007) Summaries of much of this work can be found in Pecher & Zwaan (2005).

7. Page 2 of 2 Sync in postural sway emerges during mutual conversation (Shockley, Santana, & Fowler, 2003) Sync in eye movements at shared scene emerges during conversation (Richardson & Dale, 2005) Fictive motion sentences induce more directional scanning of scene (Richardson & Matlock, 2007) Motion words influence visual perception of motion, and vice versa (Meteyard, Bahrami, & Vigliocco, 2007; Meteyard et al. 2008) Direction words heard during a reach cause curvature in the trajectory (Boulenger et al., 2006; Nazir et al., 2008)

8. The Embodiment of Language Temporal Dynamics in Neuronal Population Coding Temporal Dynamics in Spoken Word Recognition Temporal Dynamics in Sentence Processing Mental Trajectories in Neuronal State Space Temporal Dynamics in Question Answering OUTLINE

9. (Tyler & Spivey, 2005)

11. Wernicke’s Area

20. candle candy

21. Population Code for “candy”

22. Population Code for “candle”

23. 0 ms

24. 100 ms

25. 200 ms

26. 300 ms

27. 400 ms

28. Population code for recognizing “candy”

29. Population code for recognizing “candle”

30. 0ms

31. 100ms

32. 200ms

33. 300ms

34. Population code for recognizing “candle” 400ms

35. 600 300 0 0 500 1000 frontal 3/4 profile profile 1/4 profile back-of-head After Stimulus Onset (ms) Cumulative Response (spikes) Perrett, Oram, & Ashbridge (1998)

36. Temporal dynamics of population coding (Rolls & Tovee, 1995) Cumulative Information (bits) (ms) If normal eye movements occur 3-4 times per second, then a population code rarely gets enough time to asymptote!

37. Multi-dimensional scaling of the 14 cells (Rolls & Tovee, 1995)

38. Trajectories in state space for recognizing faces and objects

39. Trajectories in state space as a result of free-viewing of a scene

40. McClelland & Elman’s (1986) TRACE Model of Speech Processing

41. (McClelland & Elman, 1986) Normalized Activation TRACE Model of Speech Perception 1-sum candle candy pencil penny

42. Energy Landscape Attractor Basins

43. Vector Field Attractor Basins

44. Vector Field Attractor Basins

45. Vector Field Attractor Basins

46. On the equivalence between dynamic neural patterns and a trajectory through mental state space (Onnis & Spivey, submitted)

47. Fodor’s (1983) modularity of mind has been giving way to a continuity of mind. Visual Perception (Motter, 1993; Sekuler, Sekuler, & Lau, 1997; Spivey & Spirn, 2000) Spoken Word Recognition (Elman & McClelland, 1988; Allopenna et al., 1998) Sentence Processing (MacDonald & Seidenberg, 1993; Tanenhaus & Trueswell, 1995 Chambers, Tanenhaus, & Magnuson, 2004) Conceptual Integration and Blending (Fauconnier & Turner, 2002; Coulson, 2001) Construction Grammar, eschewing the distinction between core grammar and periphery (Fillmore, 1988; Goldberg, 2003)

48. The Embodiment of Language Temporal Dynamics in Neuronal Population Coding Temporal Dynamics in Spoken Word Recognition Temporal Dynamics in Sentence Processing Mental Trajectories in Neuronal State Space Temporal Dynamics in Question Answering OUTLINE

49. Headband-mounted Eyetracking

51. Why headband-mounted eyetracking? Saccades are frequent: 3-4 times per second. Saccades are promiscuous (low threshold for execution), and thus sensitive to subtle probabilistic biases. Eye position provides a continuous measure without interrupting processing. Eye movements are largely resistant to strategic control. Headband allows ecologically valid continuous interaction between human and environment

52. Vector Field thresholds for a saccade

53. (Tanenhaus, Spivey-Knowlton, Eberhard, & Sedivy, 1996) “Pick up the candle.”

54. (Tanenhaus, Spivey-Knowlton, Eberhard, & Sedivy, 1996) “Pick up the candle.”

55. 1.0.8.6.4.2 0 Probability of Fixation (ms) (Spivey-Knowlton, 1996)

56. McClelland & Elman’s (1986) TRACE Model of Speech Processing

57. (Allopenna, Magnuson & Tanenhaus, 1998)

58. Eye-movement data 1.0.8.6.4.2 0 Probability of Fixation (ms) (Spivey-Knowlton, 1996)

59. (Reali, Spivey, Tyler, & Terranova, 2006) “Pick up the…” Saliency Map

60. (Reali, Spivey, Tyler, & Terranova, 2006) “Pick up the ca…” Saliency Map

61. (Reali, Spivey, Tyler, & Terranova, 2006) “Pick up the cand…” Saliency Map

62. (Reali, Spivey, Tyler, & Terranova, 2006) “Pick up the candle.” Saliency Map

63. Continuous Non-ballistic Movements (Spivey, Grosjean, & Knoblich, 2005)

64. Mouse-click start box at bottom (Spivey, Grosjean, & Knoblich, 2005)

65. “Click the ladle” (Spivey, Grosjean, & Knoblich, 2005)

66. 200 300 400 500 600 700 800 900 y 02004006008001000 x (Spivey, Grosjean, & Knoblich, 2005)

68. “Click the beetle” (Spivey, Grosjean, & Knoblich, 2005)

70. Graded spatial attraction toward phonological competitors visible in averaged trajectories (Spivey, Grosjean, & Knoblich, 2005)

72. The Embodiment of Language Temporal Dynamics in Neuronal Population Coding Temporal Dynamics in Spoken Word Recognition Temporal Dynamics in Sentence Processing Mental Trajectories in Neuronal State Space Temporal Dynamics in Question Answering OUTLINE

73. Incrementality in Syntax Put the apple on... Verb Noun Phrase Prep. Phrase Preposition Verb Phrase Sentence... Syntactic Ambiguity

74. In a Visual Context One-Referent Context Two-Referent Context (Tanenhaus, Spivey-Knowlton, Eberhard & Sedivy, 1995) “Put the apple on the towel in the box” “Put the apple that’s on the towel in the box” Ambiguous: Unambiguous:

75. One-Referent Context Tanenhaus, Spivey-Knowlton, Eberhard & Sedivy, 1995)

76. Two-Referent Context Tanenhaus, Spivey-Knowlton, Eberhard & Sedivy, 1995)

77. In a Visual Context ((Tanenhaus, Spivey-Knowlton, Eberhard & Sedivy, 1995)

78. Continuous Non-ballistic Reaching Movements

79. One-Referent Context Two-Referent Context “Put the apple on the towel in the box” “Put the apple that’s on the towel in the box” Ambiguous: Unambiguous: (Farmer, Anderson, & Spivey, 2007)

84. One-Referent Context, Ambiguous Sentence (Farmer, Anderson, & Spivey, 2007)

92. The Embodiment of Language Temporal Dynamics in Neuronal Population Coding Temporal Dynamics in Spoken Word Recognition Temporal Dynamics in Sentence Processing Mental Trajectories in Neuronal State Space Temporal Dynamics in Question Answering OUTLINE

93. YES NO (McKinstry, Dale, & Spivey, 2008) start “Is a thousand more than a million?”

94. YES NO (McKinstry, Dale, & Spivey, 2008) start “Should you brush your teeth every day?”

95. YES NO (McKinstry, Dale, & Spivey, 2008) start “Is murder sometimes justifiable?”

96. (McKinstry, Dale, & Spivey, 2008)

100. The Embodiment of Language Temporal Dynamics in Neuronal Population Coding Temporal Dynamics in Spoken Word Recognition Temporal Dynamics in Sentence Processing Mental Trajectories in Neuronal State Space Temporal Dynamics in Question Answering OUTLINE

101. Toward an equivalence between dynamic neural patterns and a trajectory through mental state space (Onnis & Spivey, submitted)

103. “The new form of insight can perhaps best be called Undivided Wholenesss in Flowing Movement. This view implies that flow is, in some sense, prior to that of the ‘things’ that can be seen to form and dissolve in the flow.” -David Bohm, Wholeness and the Implicate Order (1980)

104. Slides for questions:

105. A. Weights over time for alternative reaches B. Simulated reach movements Time steps X (cm) Y (cm) Relative Activation “candy” control condition cohort condition “candy” “candle” foil “book” foil reach to foil reach to target control cohort * (Spivey, Dale, Grosjean, & Knoblich, in press)

106. Quasilinguistic thought while learning racquetball

107. Multi-dimensional scaling of the 14 cells (Rolls & Tovee, 1995)

108. Temporal dynamics of population coding (Rolls & Tovee, 1995)

109. Trajectories in state space for recognizing faces and objects

110. Trajectories in state space as a result of free-viewing of a scene

111. A B Q P R X Y m n Symbolic Dynamics