1. How Speakers' Eye Movements Reflect Spoken Language Generation Zenzi M. Griffin Department of Psychology Intro to Cognitive Science University of Texas at Austin October 14, 2011

2. TIME MESSAGEMESSAGE “ blah blah blah… ” ?

3. What the eyes tell us about speaking:  when speakers prepare names for objects  which object they intend to name  possibly when they commit to an order of mention or syntactic structure

4. Griffin & Bock (2000) Psych Science

5. Movie Legend green  current fixation blue  earlier fixation size  fixation duration

7. Speakers gazed at referents in the second before naming them Griffin & Bock (2000) Psych Science The girl is kicking the boy

8. Long gazes followed order of mention  Not left-right  or salient-less salient  or agent-patient  or big-small  or human- nonhuman

9. Word selection Difficulty in selecting a name varies with number & strength of competing candidates "baby" "TV" or "television"

10. Codability Medium High Frequency Low High

11. Isolated object naming  Latencies from a large norming study (Griffin & Huitema, 1999)  Huge, additive effects of codability & frequency. SE

12. Object categorization task  Is it bigger than a piece of paper?  Codability effect ns  Frequency effect ns  Interaction ns SE YESNO

13. “The clock and the TV are above the needle” Griffin (2001) Cognition

15. “ The |clock and the TV are above the needle. ” Griffin (2001) Cognition

16. Gaze duration correlated with name difficulty Griffin (2001) Cognition

17. Gaze duration correlated with name difficulty Griffin (2001) Cognition

18. Effects of preparing 2nd name only during speech, not before See also work by Antje Meyer & colleagues on effects of word frequency, phonological priming, image degradation, etc. Codability: MEDIUM HIGH Freq: Gaze time on B 1st noun Griffin (2001) Cognition

19. Until ready to describe The crib & the limo are above the needle time Prepare then speak

20. Gaze reflects difficulty when preparing words in advance Disfluent, Multiple names Fluent, Multiple names Fluent, One name Griffin (in prep)

21. Extemporaneous vs. Prepared speech Codability: MEDIUM HIGHMEDIUM HIGH Freq: Gaze time on B 1st noun Griffin (in prep)

22. Not much looking at locations while articulating prepared speech “ The A and the B are above the C ”

23. Gazes Reflect Word Preparation 1. When describing scenes, speakers gaze at objects just before naming them. 2. Time spent gazing at an object reflects how difficult it is to prepare a name. 3. Gazes reflect when words are prepared, not articulated.

24. Why look?

25. Gazing at intended object could facilitate name retrieval leaf

26. Gazing could prevent interference from other objects uh leaf

27. No eye movements Fixation point

28. "finger" for "thumb" (half speed then real time)

29. Timing of gaze: before error = before correct Griffin (2004) Psych Science

30. (real time)

31. Speakers gazed longer at agents before calling them by inaccurate, similar names even when only fluent trials considered  Griffin & Oppenheimer (2006) JEPLMC

32. Timing of gaze offsets was similar Griffin & Oppenheimer (2006) JEPLMC

33. (real time) Speakers even gazed at referent when a better match to name was visible Griffin & Oppenheimer (2006) JEPLMC

34. (half speed then real time) Speakers also gazed at referent when using a novel word "blick" Griffin & Oppenheimer (2006) JEPLMC

35. Name-related eye movements aren't necessary & do not seem to facilitate name preparation 1. Speakers can describe simple scenes without moving their eyes. 2. Gazes before speech errors are similar to those before correct names. 3. Speakers gaze at referents before lying about them.

36. How early must speakers commit to order-of-mention or syntactic structure?

37. AB C “The A and the B are above the C”

40. “The |woman is throwing a dog a bone.” Referent for

41. Order of mention/ structure choice "A woman throws a bone to a dog." a dog a bone." theme-recipient recipient-theme

42. 1st presentation Director/Confederat e Is a man drinking coffee? Matcher/Participant No match.

43. 2nd presentation: Biasing Question Director/Confederat e Matcher/Participant Is a tall woman throwing a bone to a dog? Yes, a tall woman is throwing a bone to a dog.

44. Biasing question like having pre- planned order: no uncertainty about order of mention "Yes, a woman is throwing a bone to a dog"

45. (real time)

46. 2nd presentation: Neutral Question Director/Confederat e Matcher/Participant Is a tall woman swimming? No, a tall woman is throwing a bone to a dog.

47. Emergent argument ordering: "Yes, a woman is throwing a bone to a dog"

48. Dependent measure: gaze shifts "Yes, a woman is throwing a bone to a dog"

49. (real time, half speed)

50. More shifts between theme & recipient during articulation of 1 st noun phrase shifts + SE Match, awoman is throwing a bone to a dog Griffin, Garton, & Mouzon (in prep)

51. More shifts/sec during 1 st NP shifts/sec + SE Match, awoman is throwing a bone to a dog Griffin, Garton, & Mouzon (in prep)

52. Conclusion  Speakers may decide order of mention & syntactic structure as needed and still be fluent.  However, additional experiments that manipulate bias in other ways suggest that time of decision varies with other factors.

53. What the eyes tell us about speaking:  when speakers prepare names for objects  which object they intend to name  possibly when they decide order of mention or syntactic structure

54. Other current projects  Tougher tests of whether eye movements during production ease cognitive load (in collaboration with Susan Goldin-Meadow & Susan Wagner-Cook)  Do eye movements to empty space during production reflect metaphorical use of space (in collaboration with Daniel Cassasanto) ?

55. Thanks to Kathryn Bock Daniel Oppenheimer Daniel Spieler Azucena Rangel Kristin Garton Elisa Lawler Edna Chavira Ellen Hamilton Sonia Mouzon Amanda Silverio Justin Storbeck Eye-speech movies, preprints, & posters available at http://oak.psych.gatech.edu/~zgriffin/