1. Sandy Zerkle Cognitive Tea Talk Series April 7, 2016
Discourse attention during utterance planning affects referential form choice
Sandy Zerkle
Cognitive Tea Talk Series
April 7, 2016

2. People Speak in Context
Introduction Methods Results Discussion
People Speak in Context
When people talk: we don’t talk about events in isolation, rather we link our sentences together to communicate successfully
We can use linking devices to mark this connection to our listener
SO something that we ALL know about language is that PEOPLE SPEAK IN CONTEXT!
When people talk, we don’t really talk about events in isolation right? Instead, we LINK our sentences together in order to communicate successfully and MORE EFFICIENTLY
We can use linguistic LINKING DEVICES to mark this connection to our listener
For example, pretend you are telling a story of these events to your friend. First you say this: “The Duke received a paining from the Duchess.”
The next thing that you might say might be: “AND THEN HE THREW IT IN THE CLOSET.”
Discourse Context:
The Duke received a painting from the Duchess.
Speaker 1:
“and then he threw it in the closet”

3. Linking Devices are Used
Introduction Methods Results Discussion
Linking Devices are Used
Discourse Context:
The Duke received a painting from the Duchess.
Speaker 1:
“and then he threw it in the closet”
This example used a few LINKING DEVICES that connect the CURRENT UTTERANCE with the PAST CONTEXT:
- Connectors are words like AND and THEN that allow us to CONJOIN statements together
- Pronouns like HE and IT tend to be used in discourse contexts where the referent is GIVEN, SALIENT, and ACCESSIBLE
- ZEROS are another type of reduced referential expression that act like pronouns but you don’t actually say it: The Duke received a painting from the Duchess and threw it in the closet.
- Another linking device is WORD ORDER, where you might utilize the same word order as the previous sentence for ease of processing (you would be LESS likely to say “then the trash can that’s sitting in the closet got a picture thrown into it.” – doesn’t really follow well)
Emphasis in PROSODY is another way to connect current and past utterances, by making the statements FLOW and SOUND like a single sentence (like I did with the Zero example)
IN THE CURRENT STUDY, WE’LL BE LOOKING SPECIFICALLY AT CONNECTORS AND PROZEROS
Linking devices: connect utterances with past context
Connectors
Pronouns/Zeros
Word Order
Prosody
(Chafe, 1976, 1995; Givon, 1983; Ariel, 1990, 1996; Brennan, 1995; Arnold, 2001, 2008)

4. Introduction Methods Results Discussion
People Vary
Speaker 1:
“and then he threw it
in the closet”
Speaker 2:
“The Duke threw the painting into the closet”
Discourse Context: The Duke received a painting from the Duchess.
Another thing that we know about language is that PEOPLE VARY in the way that we speak!
We all don’t say the exact same things, even when talking about the same concepts and in the same situation
For Example: Another speaker might choose to say: (ROBOT) “The Duke received a painting from the Duchess. The Duke threw the painting into the closet”
Sounds kind of weird right? Speaker 2 doesn’t really have to pay attention to the context (just describing single event in picture), yet his utterance still makes sense – it’s just not AS connected as Speaker 1 is
it’s INTERESTING that people seem to VARY in their use of the discourse context! This doesn’t seem random, so we want to know WHAT DRIVES THIS VARIATION?

5. Individual Variation in Reference Form
Introduction Methods Results Discussion
Individual Variation in Reference Form
In 3 similar language production studies in our lab: see individual variation in reduced form (pronouns/zeros) use across entire task
We have some EVIDENCE for individual variation in reference form choice:
In 3 similar studies from our lab looking at pronoun production, we see VARIATION by PARTICIPANT in pronoun use across the entire task
Some participants in one of Elise Rosa’s dissertation experiments used very FEW for critical trials in her task, but some used quite a lot
AND in both of my first year project experiments we also got a WIDE range of variation in the amount of reduced forms that people use throughout the experiment.
Rosa & Arnold (under review), Zerkle, Rosa, & Arnold (CUNY 2015)

6. Variation in use of the context can affect referential form choice
Introduction Methods Results Discussion
Variation in use of the context can affect referential form choice
Pronouns/zeros require the speaker to utilize the context and recognize the link with prior mentions and events, signal to listener
Speaker 1: Context-Using
Descriptions/names allow the speaker to formulate an event description without consideration of its relation to prior events
Speaker 2: Context-Ignoring
Speaker 1:
“and then he threw it
in the closet”
Speaker 2:
“The Duke threw the painting into the closet”
So we think that VARIATION in use of the context can affect REFERENTIAL form choice:
Pronouns/zeros require the speaker to utilize the context and recognize the LINK with prior mentions and events, and they signal this link to the listener
This is how Speaker 1 talked, and we’re going to call this CONTEXT-USING
On the other end of the continuum, using DESCRIPTONS or noun phrase names allow the speaker to formulate a description of an event WITHOUT considering its relation to prior events
This is like how Speaker 2 talked, and we’re going to call this CONTEXT-IGNORING
This IGNORING is probably not conscious or intentional, but these participants ARE using a different global strategy in their language production patterns – which we will investigate HOW and WHY…

7. Introduction Methods Results Discussion
WHY do people vary?
This is a larger question, the current study doesn’t characterize individual differences specifically (as other cognitive/psycholinguistic studies do)
HOW does variation in referential form choice/use of the context relate to the way utterances are planned?
Planning:
Concept/message -> word -> sounds --> sentence structure
If complex, planning takes longer = longer latency to begin speaking
So the question of WHY people vary is a pretty broad question, and the current study doesn’t characterize individual differences (at least as classic cognitive or psycholinguistic studies might)
Since this is the first study looking at this issue, a better question to start might be HOW does this variation appear behaviorally:
Specifically we want to see HOW does the variation in reference form choice -and therefore use of the context- relate to the way UTTERANCES ARE PLANNED?
By planning, we mean the process that it takes to produce each and every utterance that you make. It starts as a concept or message, then you find the lexical word, the sounds to make with your mouth, and this continues through to sentence structure
If the utterance is complex, then planning takes longer. This is typically measured with a LATENCY to begin speaking (HOW LONG IS YOUR PAUSE), which might be LONGER for complex things

8. Does utterance planning reflect this variation?
Introduction Methods Results Discussion
Does utterance planning reflect this variation?
GOAL: use eyetracking to study utterance planning
When does response planning begin? Tightly related to where people are looking
Empirical evidence:
Griffin & Bock (2000): we fixate on objects less than a second before naming (incremental planning)
Downside: not looked at within a discourse context!
Another way to measure utterance planning is with EYETRACKING
The question of When does response planning begin is TIGHTLY related to where people are looking moment-by-moment
Some empirical evidence for this comes from Griffin and Bock: They showed that speakers fixate on objects less than a second before naming it, supporting mechanisms for incremental lexical encoding
In their task, Speakers typically described scenes this these ones with ACTIVE constructions: “The mouse is squirting the turtle with water” and “The turtle is squirting the mouse with water”
their participants looked FIRST at the character they mentioned FIRST, and then looked SECOND at the character they mentioned SECOND (step by step attention/planning/uttering)
So speakers don’t HAVE to be looking at something in order to be able to name it, but it HELPS with planning if your VISUAL ATTENTION is fixated on the thing that you are trying to access and then NAME
This measure is characterized by the relative proportion of looks to each character within a time window
Now the downside to this study is that it was done WITHOUT a rich discourse context, so it is not a very good model of how speech works naturally in the world.
We’d like to investigate this further!

9. Planning Measures NEW PARADIGM:
Introduction Methods Results Discussion
Planning Measures
NEW PARADIGM:
Allows for a structured story to unfold, requires speakers to develop a rich discourse representation of each event
Utterance planning within a context:
Utterance timing reflects how complex the utterance is
Harder to pin down in latency measure, because conceptual planning might begin earlier for some
Eyetracking reflects attention
Do fixations reveal planning in a complex multi-utterance situation? Do fixation patterns reflect individual variation?
These are open questions!
So we’re hoping to use these findings to create our own NEW paradigm to show that Eyetracking and Utterance Planning are related
Our paradigm allows for a structured story to unfold, which requires speakers to develop a rich discourse representation of each event in order to successfully communicate
Our TWO utterance planning measures will be:
- The TIMING, which reflects how complex an utterance is.
It might be harder to pin down what’s going on in this latency measure, because planning might begin EARLEIR for some people
EYETRACKING, which reflects moment-by-moment attention
We want to extend Griffin and Bock’s findings to see if fixations reveal planning in a COMPLEX multi-utterance situation.
AND we want to see if fixation patterns in different TIME windows reflect individual variation in context use
THESE ARE OPEN QUESTIONS! Again this is the first study to attempt to look at these measures and individual differences together

10. Introduction Methods Results Discussion
Big Questions
Does utterance planning in a discourse context reflect differences in use of the context - as indicated by decisions about reference form (he vs. the Duke)?
Will our paradigm/planning measures reflect these differences?
This has implications in mechanisms underlying pragmatics disorders (autism), distracted speech
So here is our MAIN research question: Does utterance planning within a discourse context reflect individual differences in use of the context –
as indicated by decisions about reference form (specifically that first mention of a character – do you use a pronoun or full description name)
Our secondary question is: Will our paradigm and our specific measurements of planning REFLECT these differences?
And this has implications in informing the mechanisms underlying pragmatic disorders like AUTISM, and distracted speech

11. General Paradigm Story-telling, language production
Introduction Methods Results Discussion
General Paradigm
Story-telling, language production
General Task: See pair of pictures, hear description of first picture, must provide continuing description of second picture
53 pairs of sentences & illustrations describing a murder mystery (24 critical)
All images were shown on a computer screen
Participant told they are a tabloid photographer, viewed the entire story (no linguistic input), then discussed each pair with a “detective” to help solve the crime
Builds context within each trial, and within entire story
So this is the general paradigm that we used:
IMPORTANTLY: This builds context within EACH TRIAL, and also within the ENTIRE STORY

12. Original Design: Elise Rosa dissertation exp. 4
Introduction Methods Results Discussion
Original Design: Elise Rosa dissertation exp. 4
Lady Mannerly handed a painting to Sir Barnes.
And then he threw it away.
Lady Mannerly handed a painting to Sir Barnes.
The original design of this task comes from Elise Rosa, a cognitive student in our lab who graduated last year, where :
First the participant was shown an entire murder mystery story in pairs of pictures on a computer screen (53 pairs).
Then a LIVE “detective” confederate experimenter came in and told the participant that she needs help going through these pictures to solve the crime.
The detective described the first picture, and then the participant had to describe the second picture.
All critical sentences used a TRANSFER VERB depicting a transfer of object between two characters. The one who starts with the object is the SOURCE, and the one who ends with the object is the GOAL
This design allowed experimental control over the grammatical structure of the first sentence (who is in Subject/Nonsubject positions), and the TYPE of continuation (which character would be mentioned next, GOAL or SOURCE).
Example trial:

13. Expected Effects of Linguistic Context
Introduction Methods Results Discussion
Expected Effects of Linguistic Context
Rosa Exp.4: significantly more pronouns/zeros for Subject continuations, Goal continuations
So these are the effects of this linguistic manipulation:
It was designed to test effects of Predictability, and she found that speakers prefer to use reduced forms for salient/accessible referents like grammatical SUBJECTS and Semantic role GOALS
So this is looking at TRIAL variation, but that’s not going to be the focus here. Instead, we’re interested in the INDIVIDUAL variation patterns that people show ACROSS their entire task
These effects are not the focus here, because we’re interested in individual variation, not trial variation
(Rosa & Arnold, under review)

14. Current Paradigm Design
Introduction Methods Results Discussion
Current Paradigm Design
In order to design an experiment utilizing eyetracking and for better control of timing, a critical change was made:
Detective is no longer a live confederate, instead it is a recorded voice played over headphones

15. Current Paradigm Design
Introduction Methods Results Discussion
Current Paradigm Design
For each trial, both pictures appeared, then a 500ms delay (Preview period)
Then a recording of the “detective” describing the first panel of a scene played (control for gram. role):
A) “The Duchess handed a painting to the Duke”
B) “The Duke received a painting from the Duchess”
For each trial…....
Both pictures were shown for the entire duration of the trial, which gives the speaker a chance to look ahead (or remember) which character they will have to talk about when it’s their turn to speak (the continuation)

16. Current Paradigm Design
Introduction Methods Results Discussion
Current Paradigm Design
3. Participants’ task was to describe the continuing action in the second panel, and then click to move on to the next trial
“and {Ø / He / The Duke} threw it in the closet”
Examined variation in:
pronouns/zeros
connectors (and, then, so)
latency to begin speaking fluently
They could say anything they wanted about the second panel, but we only included trials where the first character they mention is the Target character.

17. Current Paradigm Design
Introduction Methods Results Discussion
Current Paradigm Design
Eyetracking analysis:
3 Fixation Regions:
Panel 1-Target, Panel 1-Non-Target, and Panel 2-Target (empirical logit calculated for each (Barr, 2008))
Panel 1
Target
Panel 2
Target
Panel 1
Non-Target
AGAIN, our goal for this study was to implement the original paradigm with EYETRACKING, so
We looked at 3 regions of fixations in each trial for our eyetracking analyses:…..

18. Did Individuals Vary? N = 37 Two Response Patterns:
Introduction Methods Results Discussion
Did Individuals Vary?
N = 37
Two Response Patterns:
Context-Ignorers (24 ppts) Context-Users (13 ppts)
(must use at least one pro/zero)
- Only descriptions % pronouns/zeros
- 1% connectors % connectors
- Almost no pronouns in Fillers - Pronouns in Fillers
- Longer latency - Shorter latency
So here’s our RESULTS! First we wanted to see IF INDIVIDUALS VARIED:
We identified two response patterns and PUT people into TWO DIFFERENT GROUPS:
24 ppts ONLY used descriptions to refer to the Target (in critical trials)– call Context-Ignorers
the other 13 ppts used some amount of reduced forms – call Context-Users.
The definition of this group is whether they used at least ONE pronoun or zero
They appeared to be considering the discourse context as they planned their utterance: (go through list)
Context-Ignorers: Even on the subset of Filler trials where only one character acts in both pictures, these participants used majority descriptions (9 pronouns out of 252 trials), further emphasizing this distinction in use of the context throughout the entire experiment.

19. How do Context-Users Use Context?
Introduction Methods Results Discussion
How do Context-Users Use Context?
Connector Use (and, and then, so):
Reduced Form Use:
Context-Users: More pronouns/zeros
For Subject continuations
Context-Users: More connectors than Context-Ignorers * *
So we have these two different groups, and now we want to CHECK that the Context-Users are using the Context in the ways that we would EXPECT them to trial-by-trial:
we replicated previous findings that speakers prefer reduced forms for salient/accessible referents like Subjects and Goals:
SIG SUBJECTS
TREND GOALS
Context-Users used more connector words (and, and then, so) than Context-Ignorers (p = ).
Within the Context-Users, SIG SUBJECTS
Connector words help signal the relation between two utterances.
The fact that connectors were virtually absent from the Context-Ignorers’ speech HIGHLIGHTS our conclusion that they were just focusing on the response in isolation, and ignoring its relation to the context.
“The Duke received a painting from the Duchess”
“The Duchess handed a painting to the Duke”
All stats from mixed effects models (SAS proc glimmix and proc mixed)

20. Do Fixations Reflect Planning?
Introduction Methods Results Discussion
Do Fixations Reflect Planning?
Eyetracking Analysis
4 Time Windows:
Preview: First 700ms ( ) when participant sees pictures but no linguistic input yet
Detective Sentence: when detective sentence is playing over headphones (different length for each trial)
Latency: duration of time after detective sentence stops and before participant’s fluent speech starts
Response: when participant utters their continuation sentence
Next we want to see if FIXATIONS REFLECT PLANNING, and if IT WILL HELP DISTINGUISH THE GROUPS
For our eyetracking analyses, we looked at 4 time windows within a trial:…

21. Fixations Reflect Planning
Introduction Methods Results Discussion
Fixations Reflect Planning
Preview
Detective Sentence
Latency Response
Onset of picture pair
Onset of DS+ 200ms
Onset of Response
YES, in this task Fixations DO reflect planning!!
Four time windows of a trial, averaged over all items, first showing just Context Users
This shows AVERAGE LOOKS for each of the three characters (point out colors)
On the x-axis is time, starting at the beginning of the trial for the Preview/Detective Sentence graphs – showing onset of pictures, onset of DS,
THEN centered around the onset of participants’ response in the Latency/Response graphs.
Grayed out boxes in Latency/Response graphs indicate that the detective sentence is still playing,
and Latency window is the average of the group
THEN show for Context-Ignorers – looks pretty similar:
Everyone starts out looking at Panel 1 a lot, equals out as listening to DS, Panel 1 then drops off and most looks to Panel 2 before and during utterance
but there are some significant differences between the groups!
Because we were interested in planning effects, the most interesting windows are the Preview period (when planning of Target begins already) and the Latency period (when they’re doing more PROXIMAL planning of utterance)
Perhaps unsurprisingly, there were no strong trends within the Detective Sentence window.
This could be because the time window is long, and participants tend to be looking at all three characters roughly equally as they are listening to the Panel 1 characters being named and planning to talk about the Panel 2 Target.
We found effects in these time windows supporting both evidence for linguistic planning by all speakers, and differences in attention by the Context Use groups.
I will now explain these effects separately:
“The Duchess handed a painting to The Duke”

22. Fixations Reflect Planning
Introduction Methods Results Discussion
Fixations Reflect Planning
Preview: Both groups initially fixate more to the Panel 1 Target than the Panel 1 Non-Target (combined Goal/Source – no Thematic Role effect)
So first we found that ALL speakers begin planning at the VERY beginning of the trial:
In the PREVIEW period (FIRST 700ms (X AXIS) – all speakers fixate more to the Panel 1 Target than the Panel 1 NON-Target
This implies that even in this very EARLY period, everyone was LOOKING MORE AT and PLANNING which character they will talk about in their upcoming utterance, either based on knowledge from seeing the entire story beforehand or their peripheral vision to Panel 2. *

23. Fixations Reflect Planning
Introduction Methods Results Discussion
Fixations Reflect Planning
Response: Both groups fixate significantly more on Panel 2 Target in second before naming it (Griffin & Bock, 2000)
We also found that ALL speakers show more PROXIMAL planning right before their response:
In Both Groups, speakers fixate more on the Panel 2 Target character in the SECOND before naming it (than in the entire DS window)
This supports evidence from Griffin and Bock (2000) that we tend to look at what we are about to talk about

24. Do planning measures reveal group differences?
Introduction Methods Results Discussion
Do planning measures reveal group differences?
Context-Users respond faster than Context-Ignorers: *
NEXT we want to see if PLANNING MEASURES (timing and eyetracking) reveal GROUP DIFFERENCES:
As I said before, saw a group difference in LATENCY, where
Context-Users have a SHORTER latency to begin speaking fluently than Context-Ignorers (1461 ms vs ms, p = ).
This indicates that Context-Users are FASTER to plan their utterances overall, which might mean that they are systematically paying MORE attention to the task as a whole than the Context-Ignorers are, and thus are QUICKER to speak their response
This might seem surprising, since consideration of the context could imply a GREATER degree of information processing.
However, trials with SHORTER latencies are also those where speakers are likely to have engaged in a GREATER degree of pre-planning earlier in the trial.
If the speaker were to think about the response while ALSO processing the context sentence, it would require having BOTH events co-activated.
Co-activation may support the activation of relations between events, increasing the use of linguistic devices to mark coherence.

25. Group Differences * * Preview: Introduction Methods Results Discussion
Context-Users look to the Panel 1 Target more than the Context-Ignorers
Context-Ignorers look to the Panel 2 Target character more than the Context-Users * *
Next we want to see if FIXATIONS reveal group differences:
In the Preview Period: The Context-Users looked more at the characters in the context panel, WHILE the Context-Ignorers were looking more at the target panel in this period.
This shows up in two analyses:
Users look to the Panel 1 Target more than Ignorers (p=0.032),
and Ignorers look to the Panel 2 Target character more than Users (p=0.0004).
This implies that the Context-Users were paying more attention to the characters in the first context panel, especially the one they would soon talk about (the Target),
while the Context-Ignorers tended to look faster to the second panel, and not examine the context panel as much.

26. Introduction Methods Results Discussion
Group Differences
Response: Context-Users look at the Panel 2 Target character in the 500ms before speaking more than the Context-Ignorers
We also see group differences in fixations around the RESPONSE window:
This graph is showing looks to the Panel 2 Target, and
Context-Users looked more at this character in the 500ms before speaking than the Context-Ignorers do (p=0.0109)
This is a POST HOC SELECTION OF WINDOW, BUT there is a VISUAL TREND OVERALL throughout the latency and response periods (just qualitatively significant in this window here)
This suggests that the Context-Users may have been more SYSTEMATIC in their fixation pattern, examining the context panel FIRST, and then turning their ATTENTION to the response panel.

27. Introduction Methods Results Discussion
Paradigm works! We can understand group differences in terms of planning measures
Individuals differed: Context-Users used more reduced forms and more explicit connector words than Context-Ignorers.
Planning metrics predicted individual strategy:
Context-Users:
Had shorter latency, possibly doing more pre-planning.
Looked more at the Panel 1 Context picture during Preview period, possibly doing more initial processing of the first event (context).
Looked more at the Panel 2 Target picture in 500ms before the Response period, revealing more systematic planning.
SO What have we learned from this study? Well first, we know that our paradigm works! We can now understand group differences in context use in terms of planning measures
We found individual diffs, where Users use more pronouns and zeros and more explicit connector words
Our planning measures predicted individual strategy in context use:
The USERS:
Had shorter latency = doing more pre-planning of utterance EARLIER ON IN THE TRIAL
Looked more at Context panel in PREVIEW period = doing more initial processing of that first event
Lookes more at Target panel right before RESPONSE = more systematic planning to refer to this character by this group as a whole

28. Discussion Fixations revealed planning in both groups:
Introduction Methods Results Discussion
Discussion
Fixations revealed planning in both groups:
Everyone has initial looks to Panel 1 Target, and Panel 2 Target looks in second before response initiation.
Next questions: Why did this particular task show a different pattern of pronoun use? How can we get speakers to produce more pronouns in this task?
This is motivated by our original question, which is to investigate the planning effects of thematic role predictability (Goal/Source) on reference form choice
Need more trial-wise reference form variation for this
Perhaps due to missing social/conversational aspect of live Detective
Experiment 2 in progress! Emphasizing “stortytelling”
We also found that Fixations revealed PLANNING in BOTH groups:
Everyone has INITIAL looks to the Panel 1 Target - because they know that’s who they’ll need to talk about, and
Everyone looks at the Panel 2 Target right before their RESPONSE - because looking at what you’re naming HELPS with planning and production
Our next questions are: …

29. See jaapstimuli.web.unc.edu for stimuli used!
Thanks!
Dr. Jennifer Arnold
Dr. Elise Rosa
Kathryn Weatherford
Research Assistants:
Jacob Pascual
Jenna Roller
Brianna Torres
Kirsten Bubak
Stimuli Design:
Liz Reeder
Ana Fetterman
S. Adam Smith
See jaapstimuli.web.unc.edu for stimuli used!