1. Language Embedded in Systems of Perception and Action Rolf A. Zwaan Florida State University

2. Experiential Traces

3. “referential” linguistic

4. Experiential Associations “intra-referential” intra-linguistic

5. Experiential Associations cross-domain

6. Hebbian Learning Co-occurrence primary factor in the forming of associations Co-occurrence primary factor in the forming of associations Intra-linguistic; Intra-linguistic; Intra-referential; Intra-referential; Cross-domain (grounding). Cross-domain (grounding).

7. Experiential Traces Linguistic Traces Linguistic Traces Auditory Auditory Motor Motor Visual Visual Tactile Tactile Referential Traces Visual Motor Auditory Olfactory Tactile Somatosensory

8. Capturing Second-order Correlations r 2 =.30

9. r 2 =.65 Capturing Second-order Correlations

10. Utility of Linguistic Traces/Associations Additional layer of associations. Additional layer of associations. May enhance “fluency.” May enhance “fluency.” “Referential bootstrapping.” “Referential bootstrapping.”

11. Referential Bootstrapping Name this animal: Name this animal: What has horns like a giraffe, a deer head, a horse neck, and legs like a zebra? http://www.westerville.k12.oh.us/ZOO/Orlich/pages/Okafrtpg.htm What has horns like a giraffe, a deer head, a horse neck, and legs like a zebra? http://www.westerville.k12.oh.us/ZOO/Orlich/pages/Okafrtpg.htm

12. Referential Bootstrapping

13. Resonance Experiential traces “resonate” with linguistic input. Experiential traces “resonate” with linguistic input. Visual resonance. Visual resonance. Motor resonance. Motor resonance. The role of resonance in comprehension. The role of resonance in comprehension.

14. Predictions Prediction 1 Prediction 1 Language comprehension affects subsequent perceptual performance. Language comprehension affects subsequent perceptual performance. Prediction 2 Prediction 2 Incidentally acquired visual representations affect subsequent language processing. Incidentally acquired visual representations affect subsequent language processing. Prediction 3 Prediction 3 Language comprehension affects concurrent motor performance. Language comprehension affects concurrent motor performance.

15. Comprehension Affects Perceptual Performance (Zwaan, Stanfield, & Yaxley, 2002) The ranger saw the eagle in the nest. [SAW, RANGER, EAGLE][IN, EAGLE, NEST] The ranger saw the eagle in the sky. [SAW, RANGER, EAGLE][IN, EAGLE, SKY]

16. The ranger saw the eagle in its nest. The ranger saw the eagle in the sky.

17. Picture Recognition and Naming Latencies (Zwaan, Stanfield & Yaxley, 2002) RT (ms)

18. Language may Involve Dynamic Representations (Zwaan. Madden, Yaxley, & Aveyard, 2004) The midfielder passed the ball to you.

23. Zwaan, Madden, Yaxley, & Aveyard, 2004

24. Prediction 2 Incidentally acquired visual representations affect subsequent language processing. Incidentally acquired visual representations affect subsequent language processing.

25. The Visual-Memory Paradigm (Aveyard, Zwaan, Radach, & Vorstius, in preparation) Memory Phase Memory Phase Word-picture verification (4 times) Word-picture verification (4 times) Reading Phase (ostensibly unrelated) Reading Phase (ostensibly unrelated) Eye movements tracked Eye movements tracked

26. Memory Phase eagle

27. Memory Phase

28. Reading Phase The ranger was in the park on his morning round. It was a bright day. In the sky there was an eagle, soaring above the prairie.

29. First Fixations

30. Gaze Durations

31. Visual World Visual Memory Situated Situated Visual Search Visual Search Auditory Presentation Auditory Presentation DV = eye fixations during display viewing DV = eye fixations during display viewing Re-situated Memory Search Visual Presentation DV = eye fixations during reading

32. Prediction 3 Motor traces are activated by linguistic input. Motor traces are activated by linguistic input.

33. Action Compatibility Effect (Glenberg & Kaschak, 2002) Sensibility judgments Sensibility judgments Close the drawer. Close the drawer. Andy delivered the pizza to you. Andy delivered the pizza to you. Liz told you the story. Liz told you the story. Break the cloud. Break the cloud.

34. Action Parameters and Motor Resonance Actions have various parameters. Actions have various parameters. Direction. Direction. Force. Force. Manner (e.g., open hand vs. closed hand). Manner (e.g., open hand vs. closed hand). Speed. Speed. Duration. Duration. Which action parameters exhibit resonance in language comprehension? Which action parameters exhibit resonance in language comprehension?

35. Measuring Response Force

36. 10% T 90% A Atmospheric pressure = 100 kPa

37. Response Force (Zwaan & Taylor, in prep.) Sensibility judgments. Sensibility judgments. Verbs. Verbs. He pushed the car. (high) He pushed the car. (high) He started the car. (low) He started the car. (low) He admired the car. (no) He admired the car. (no) Nouns. Nouns. He broke the baseball bat. (high) He broke the baseball bat. (high) He broke the bread. (low) He broke the bread. (low) He broke the promise. (no) He broke the promise. (no)

38. Response Force: Verbs Response Amplitude (kPa)

39. Response Force: Nouns Response Amplitude (kPa)

40. Manner of Action Sustain vs. Release Sustain vs. Release He gripped the hammer. He gripped the hammer. He dropped the hammer. He dropped the hammer. Predictions Predictions Sustain faster than Release (ACE). Sustain faster than Release (ACE). Sustain longer than Release (ACE). Sustain longer than Release (ACE).

41. Sustain vs. Release Response Speed

42. 10% Sustain vs. Release Response Duration

43. Sustain vs. Release Response Duration >10%

44. Motor Resonance What is the role of motor representations in language comprehension? What is the role of motor representations in language comprehension? Instrumental (Glenberg) Instrumental (Glenberg) Ornamental (many skeptics) Ornamental (many skeptics) Useful/necessary in specific cases (Jackendoff) Useful/necessary in specific cases (Jackendoff)

45. Motor Resonance What is the role of motor representations in language comprehension? What is the role of motor representations in language comprehension? Useful/necessary when actions are Useful/necessary when actions are Concrete Concrete Punctate Punctate Simple Simple Let’s first ask: How is motor resonance modulated by linguistic input? Let’s first ask: How is motor resonance modulated by linguistic input?

46. How is motor resonance modulated by linguistic input? ( Zwaan & Taylor, 2006, JEP General) Reading by rotating. Reading by rotating. Every 4 deg. new frame. Every 4 deg. new frame. He /realized /that /the music /was /too loud/so he /turned down/the /volume. He /realized /that /the music /was /too loud/so he /turned down/the /volume. 60 psychology undergraduates. 60 psychology undergraduates.

47. Reading-by-Rotation

48. Is Motor Resonance Short-lived or Focus-bound? (Taylor & Zwaan, in prep.) Perspective change. Perspective change. After/lighting/the candles/for the/romantic/evening,/he/dimmed/the/lights. Maintaining focus on the action. Maintaining focus on the action. He lit/the candles/for the/romantic/evening./He noticed/the bright/lights,/which he/dimmed/slowly. Reading-by-rotating. Reading-by-rotating.

49. Motor Resonance is Focus-bound

50. Strong Test of Linguistic Focus Hypothesis Action-modifying vs. subject-modifying adverbs Action-modifying vs. subject-modifying adverbs … which he turned down slowly. … which he turned down slowly. …which he turned down obediently. …which he turned down obediently. Stay tuned. Stay tuned.

51. Conclusions & Outlook Language is embedded in systems of perception and action. Language is embedded in systems of perception and action. Rapid exchange of information. Rapid exchange of information. But it needs to be determined to what extent it is in bed with these systems. But it needs to be determined to what extent it is in bed with these systems. Perceptual & motor resonance ornamental or instrumental? Perceptual & motor resonance ornamental or instrumental?

52. Acknowledgements Grants from the National Science Foundation (NSF) and the National Institute of Mental Health (NIMH) Grants from the National Science Foundation (NSF) and the National Institute of Mental Health (NIMH) Graduate students (past & present) Graduate students (past & present) Rob Stanfield, Carol Madden, Rich Yaxley, Mark Aveyard, Larry Taylor Rob Stanfield, Carol Madden, Rich Yaxley, Mark Aveyard, Larry Taylor Postdoctoral Fellows (past & present) Postdoctoral Fellows (past & present) Barbara Kaup, Dave Therriault, Johanna Kaakinen, Tobias Richter Barbara Kaup, Dave Therriault, Johanna Kaakinen, Tobias Richter Faculty Collaborators Faculty Collaborators Mike Kaschak, Ralph Radach, Katinka Dijkstra Mike Kaschak, Ralph Radach, Katinka Dijkstra