Watching the eyes when talking about size: An investigation of message formulation and utterance planning Sarah Brown-Schmidt, Michael K. Tanenhaus Presentation by Olatz Mendiola Ling 5555 November 17, 2010

Introduction How do speakers plan and update messages that are translated into utterances during language production? How are message formulation and utterance planning coordinated? Speakers often update messages on the fly based on new information.  Message formulation and utterance planning are mixed in time and must communicate with one another at a relatively fine temporal grain.

Introduction Levelt & Maassen (1981)  Latency difference evidence that the unit of advance planning includes each of the lemmas in the subject noun phrase. Meyer (1996)  Dutch speakers perform a naming task with auditory distractors.  Naming latencies were delayed by a distractor that was phonologically related to the first noun in NP but not the second word in the NP.  Distractors semantically related to either 1st or 2nd noun decreased latencies. Schriefers &Teruel (1999)  Presented phonological distractors to German speakers in a naming task.  Phonological distractors related to 1st syllable of 1st word facilitated naming times, no effect found for distractors related to 2nd word and distractors related to 2nd syllable of 1st word had a weak effect.  In some cases, articulation may begin before the 1st phonological word is entirely planned. Only 1 st word is prepared before speech onset

Introduction The degree to which speakers pre-plan a complex phrase is highly variable.  Amount of advance planning varies depending on the task and type of utterance  Planning is partially under control of speaker. Little work has addressed how messages are prepared Griffin and Bock (2000)  Participants described simple events (woman shooting a man…)  The sequence of eye movements reflected the order of constituents in the utterance.  Speakers looked at picture objects about 800ms to 1s before naming them.  This eye-voice lag is similar to the time it takes to initiate naming an object in isolation, suggesting that the eye-voice delay reflects word preparation. Difficult?

Introduction Timing of eye movements could provide insight into interplay between message formulation and utterance planning. Work investigating gaze-speech link may typically conflate message formulation with utterance planning. How disentangle message formulation form utterance planning?   creating situations in which the speaker, while in the process of planning or producing an utterance, encounters new information that requires revising the message.  If eye movements can be used to infer when the speaker 1st encounters that information, then the timing between the uptake of the new information and the form of the utterance might shed light on the interface between message production and utterance planning. Brown-Schmidt Tanenhaus (2006 )

Present Study. Experiment 1 18 pairs of participants Seated at different computers. 5 feet separation No eye contact Subject X  Eye tracker + headset microphone Subject Y  Headset microphone

Listener’s perspectiveSpeaker’s perspective

Present Study Hypothesis 1  Fixation to the contrast will predict use or non-use of a size adjective. Higher rates of modification when contrast has been fixated. Hypothesis 2  There is a systematic relationship btw timing of 1 st look to the contrast and the form of the utterance. Hypothesis 3  Timing of looks to the contrast and use of a pre-nominal vs. post-noun repair for descriptions of complex and simple shapes should provide insight into the size of message planning units, and how repairs to these units affect utterance planning.

Results Manipulations successful in eliciting expected forms: 1. Size adj. typically used only when contrast present in display. 2. Pre-nominal adj. used for Simple Shapes. Adj. in post-nominal PP used for Complex Shapes.

Results 1. Use of size adjectives Contrast : size adj. on 98% of trials No contrast: size adj. on 27% of trials Contrast/Simple: size adj. on 99% of trials Contrast/Complex: size adj. on 97% of trials NoContrast/Simple: size adj. on 23% of trials NoContrast/Complex: size adj. on 31 % of trials Statistically Significant Not significant

Results 2. Utterance form  Contrast / Size adj. / Simple shape  pre-nominal  89% of trials  Contrast / Size adj. /Complex shape  post-nominal PP  73% of trials Size adj. also occurred in post-noun repairs “The triangle, uh the small one” “The square with triangles, uh small ones”  11% of trials for simple shapes  16% of trials for complex shapes

Results. Hypothesis 1  Fixation to the contrast will predict use or non-use of a size adjective. Higher rates of modification when contrast has been fixated.  Size contrast / fixated Size adj. on 99% of trials  Size contrast / not fixated Size adj. on 68% of trials

Results. Time of the first fixation to the contrast relative to onset of utterance. Form of referring expression is related to timing btw 1 st look to size contrast and onset of utterance. ANOVA:  Effect of adjective position Earlier fixation for pre-nominal adj.  Effect of complexity. Earlier fixation for references to simple shapes  Interaction btw complexity and adj. position was not significant New message elements can be added to a planned referential description during or immediately before production Supports Hypothesis 2UNEXPECTED

Results Simple shapes: Negative relationship between first fixations and adj. onset time. Complex shape: marginal effect When need for a size adjective is noticed after speech onset, preparation of the repair is delayed by preparation of the original utterance. Lag between 1 st fixation to contrast and adjective onset  Complex: 1220ms  Simple: 440ms

Results Simple shape  Prenominal 1546ms after display  Post repair 1061ms after display Complex shape  Prenominal 1434ms after display  Post repair 1390ms after display

Conclusions Fixation on size contrast predicts use of size adj. Timing of fixations predicts the form of the utterance. When preparing a modified referring expression, messages are incrementally prepared and passed onto utterance planning in units smaller than the size needed for an entire referring expression.

Towards experiment 2 2 unexpected results  600ms lag between fixation to the contrast and onset of utterance. 200ms faster than literature reports. Explanation? A small number of shapes were repeatedly used on target trials, which may reduce the demands of lexical encoding  Size adj. were used on more than 25% of trials when there was no contrast. Explanation? 25% of trials had targets with size contrast which might make inclusion of size in the description somewhat formulaic Experiment 2  Increasing target type  Decreasing proportion of target trials

Experiment 2

Results Speakers rarely used a size adj. when there was no contrast in display = 1 % Speakers used a size adj. 72% of the references on trials with contrast Proportions consistent with other studies Of the size contrast trials where adj. was used:  62% pre-nominal  37% post-noun repairs

Results Analysis was restricted to trials were there was a size contrast on same side of the river as target. Fluent and disfluent trials analyzed separately. Fluent trials  Looks to contrast predicted whether the size of target would be mentioned (Hyp 1) = Exp 1 86% of fluent trials during which speakers looked at contrast, the referring expression included a size modifier. When there was no fixation to the contrast, modifier was only used in 19 % of trials  First fixation to the contrast was systematically related to the form of the utterance. (Hyp 2) = Exp 1 The mean 1 st fixation to contrast for pre-nominal modifiers was 887ms before onset of referring expression. The mean 1 st fixation to contrast with post-noun repairs was 14ms before onset.

Results Disfluent trials  When speakers fixated the contrast adj was used on 91%.  When contrast was not fixated ajd was used 34%.  First fixation to contrast systematically related to form of utterance. First fixations were on average 528ms before onset of referring expression for utterance with pre-nominal modifiers. First fixations were on average +819ms for post-noun repair utterances

Results. Earliest fixations are associated with fluent prenominal utterance Latest fixations are associated with a post-noun repair. Consistent with H3: suggests speaker was able to use disfluency to buy time to include size in the noun phrase as a prenominla adj. rather than as a postnoun repair. =Exp 1. Relationship btw time speakers 1 st fixated the size contrast and the timing of the size adj. for postnoun repairs.

Results Time to begin speaking  Average time between the highlighting of target and NP onset < 2s  ≠ Exp1. Utterance onset time was not significantly affected by the form of the utterance  For initially fluent utterances, speakers started pre-nominally modified expressions 345 ms faster than expressions with post- noun repairs (not reliable)  For disfluent utterances pre-nominally modified expressions were only 120ms faster than utterances with postnoun repairs  Speakers uttered disfluent pre-nominally modified expressions on average 66ms faster than fluent pre-nominally modified utterances (not reliable)

General Discussion 3 results  1. Speakers typically used size adj. in a referential expression only when there was a similar object in the referential domain that differed in size  2. Use of size adj was more likely if speaker had fixated on the size contrast.  3.Timing of first fixation to contrast was related to the form of referential expression Earlier – prenominal adj Later – postnoun repair

General Discussion Preliminary support for hypotheses about interplay between message formulation and utterance planning. When an error is detected in the utterance plan after speaking has begun, initiation of a repair may be delayed by preparation of original utterance. Messages that will be mapped onto referring expressions can be constructed and passed onto utterance planning incrementally.

Eskerrik asko!