1. Thematic Role Predictability and Planning affect Word Duration
Sandra Zerkle, Elise Rosa, and Jennifer Arnold
University of North Carolina at Chapel Hill
ETAP-3: Experimental and Theoretical Advances in Prosody
University of Illinois, Urbana-Champaign, May 28-30, 2015

2. Reference Form Varies
Speakers choose between lexically reduced forms of reference (pronouns) and explicit names
John went skiing with Matt.
Then he fell down.
Then Matt fell down.
Then John fell down.
Pronunciation ranges from acoustically reduced to prominent
Then JOHN fell down.
Then J..o…h..n… fell down.
...
We know that reference form varies: particularly in two ways
Speakers can choose…..
We also know that pronunciation ranges from acoustically reduced to acoustically prominent - as we have seen in the talks today, there’s lots of ways to measure the very variable acoustic properties of these words.
How do people CHOOSE how to say these words? In this talk, I’ll be looking at the PREDICTABILITY of these types of words and asking if and how they affect these properties of language production – specifically Acoustic Reduction

3. Predictability has strong effects on acoustic reduction
Lexical Predictability
Speakers produce certain words with shorter durations/greater phonological reduction when they are predictable
Probabilistic Reduction Hypothesis: word frequency, statistical probability, context, sentence structure, discourse and semantic factors (Bell et al., 1999; Jurafsky et al., 1998, 2001), Lexical-Access- Based Model (Bell et al., 2009, 2003), Likelihood of co-occurrence in context (Lieberman, 1963), Frequency of syntactic structure (Gahl & Garnsey, 2004)
Referential Predictability
Speakers are more likely to re-mention certain entities/referents than others, making those references predictable or expected
Accessibility – likely to be re-mentioned:
Grammatical role (Arnold, 1998), probability in context (Watson, Arnold, & Tannenhaus, 2007), repetition (Lam & Watson, 2010), givenness (Fowler & Housum, 1987)
Thematic Role?
There are two different types of predictability found in the literature that guide these choices:
The first is LEXICAL Predictability, where speakers produce certain words with shorter durations/greater phonological reduction when they are predictable (as seen in the previous example)
- In terms of Lexical Predictability, there are certain properties of language that systematically guide speakers to use reduced forms
- One theory is that speakers use reduced forms because the word they are saying is more predictable in the discourse, and they don’t have to use as much emphasis as when the word is less predictable. But speakers have a CHOICE about how explicit to make an expression, and there is evidence in the literature that certain properties of language guide this choice, among other things.
For example, Bell et al. showed that FREQUENCY affects word duration by speeding lexical access. The FREQUENCY of words in the preceding and following context of a target word matter, and change the duration of that target word systematically. The durations of very frequent FUNCTION words are sensitive to the predictability of the PRECEDING word, and the durations of both content and function words are strongly affected by predictability from the word FOLLOWING them.
Another example comes from Lieberman, where the word NINE in the common maxim “a stitch in time saves nine” has LESS EMPHASIS (lower intelligibility) than in “the next number you will hear is nine”. This shows that the acoustic properties of words are affected by their likelihood of co-occurrence in context.
- All of these studies show that Lexical Predictability leads to acoustic reduction, where there is less emphasis and/or shorter duration for these words
The second type of predictability is REFERENTIAL Predictability, where speakers are more likely to re-mention certain entities/referents than others, making those references predictable or expected
- there’s a lot of evidence that certain Discourse Properties confer Accessibility, which guides a speaker’s choice of re-mentioning an entity
- For example, Watson, Arnold, and Tannenhaus showed that probability in context matters: in a tic tac toe game played between partners, moves that are PREDICTABLE (winning moves or blocking a winning move) are more accessible and have SHORTER duration than moves that are less predictable.
- These studies all show that Referential Predictability leads to accessibility and likelihood of re-mention.
So from all of this we know that predictability has strong effects on acoustic reduction.
(CLICK) But one open question in terms of Referential Predictability is about THEMATIC ROLE, and if a Semantic Bias has an EFFECT ON ACOUSTIC REDUCTION.
We might predict that it DOES, because other types of Referential Predictability do, but so far Thematic Role has only been examined in terms of pronoun use.
In our current experiments we will be looking specifically at THEMATIC ROLE predictability and asking if it affects Acoustic Reduction. For example…

4. Semantic Bias: Thematic Role
GOAL CONTINUATIONS
Sir Barnes got a picnic basket from Lady Mannerly. Sir Barnes…
GOAL source
More Predictable
SOURCE CONTINUATIONS
The chef handed a cookbook to the maid. The chef…
SOURCE goal
In these sentences, there is a transfer of an object from one character to another. Sir Barnes got a picnic basket from Lady Mannerly, The chef handed a cookbook to the maid.
If the next sentence here were to start with Sir Barnes, who has the thematic role of GOAL, these continuations have been shown to be MORE PREDICTABLE
If the next sentence here were to start with The Chef, who has the thematic role of SOURCE, these have been shown to be LESS PREDICTABLE
This makes sense, because we tend to want to talk about the character who NOW HAS the object that was transferred (GOALS). What is Sir Barnes now going to do with the basket he just got?? It’s less logical to want to talk about The Chef, because we’d rather want to know about what The Maid is doing.
In natural speech, Goals tend to get rementioned more often than Sources, and Goals tend to get pronominalized more than Sources
Less Predictable
Kehler et al., 2008; Rosa, 2015

5. Need to Control for Grammatical Role
GOAL CONTINUATIONS
Lady Mannerly gave a picnic basket to Sir Barnes. Sir Barnes…
source GOAL
More Predictable
SOURCE CONTINUATIONS
The maid took a cookbook from the chef. The chef…
goal SOURCE
In these types of sentences, it’s important to control for Grammatical Role when designing an experiment.
Now the characters are flipped in the SUBJECT and NONSUBJECT positions, but Thematic Roles stay the same. In these sentences the character who continues was in the NONSUBJECT position in the preceding sentence. (whereas on the previous slide the continuing characters were in the SUBJECT position)
This is important because there is a well-known effect that SUBJECTS tend to be reduced when re-mentioned, and we wanted to control for this GRAMMATICAL effect. In our experiments, this is controlled for across participants and items.
(notably, Kehler et al did NOT control for Grammatical role…)
Less Predictable
Kehler et al., 2008; Rosa, 2015

6. Study Goals
Does Thematic Role Predictability affect Acoustic Reduction?
What is the role of Planning?
How are Acoustic Reduction and Planning related?
Here is a roadmap for this talk, highlighting our main questions.
The first is: Does THEMATIC ROLE predictability affect Acoustic Reduction? We will answer this by looking at the DURATION of a target word
Next: What is the role of utterance planning? Predictability might be related to the time needed for a speaker to PLAN their utterance. We will answer this by looking at the LATENCY to begin speaking fluently.
Finally: we want to see How the two are related? The effects of Predictability might NOT guide Duration DIRECTLY but instead might be MEDIATED by Planning Time. This will give us insight into WHY and HOW Predictability matters.

7. Planning Effects How might Thematic Role affect Planning Time?
One way to examine planning: Latency
Hypothesis: Goals are related to easier planning
First I’ll lay out our predictions about Planning Effects.
How might Thematic Role affect Planning?
One way to examine planning is with a measure of LATENCY: the time before a speaker begins an utterance. We can’t know precisely what’s happening when a speaker hasn’t started speaking yet, but this time represents a good measure for the planning that is happening, both conceptually, lexically, and phonologically.
Our Hypothesis is that Goals are related to EASIER planning. This makes sense because we know that goals are more predictable, the get re-mentioned more, and they get pronominalized more, so that makes them relatively easy to plan and we might see a shorter latency to begin speaking for these references.

8. Planning Effects Predictions:
Pre-plan: If planning utterance is easy, all planning happens before utterance begins
If only pre-planning, expect no relationship between latency and duration
Incremental: If planning utterance is more difficult, speakers can pre-plan some but must begin speaking while still planning the rest of their utterance incrementally
Has consequences for both pre-planning AND incremental planning, expect latency and duration to be correlated
Share the Load: Ease Planning (Christodoulou, 2012)
Here are our predictions about different TYPES of planning, which Jennifer mentioned in her talk earlier but I will explain again:
One type of planning is PRE-PLANNING. If planning an utterance is easy, then all of the planning happens before the utterance begins
- If a speaker is only pre-planning, we expect no relationship between latency and duration. Long latencies would mean that it took a long time to plan the utterance, but once they get going there’s no reason to expect them to lengthen the duration of words or be disfluent
INCREMENTAL PLANNING could also happen: If planning an utterance is more difficult, speakers can pre-plan some words but must begin speaking WHILE STILL planning the rest of their utterance incrementally (word by word)
- This has consequences for both pre-planning AND incremental planning, and we expect latency and duration to be correlated
These predictions depend on whether we assume speakers are pre-planning most of their utterances, or whether they are doing some COMBINATION of pre-planning and incremental planning.
(CLICK) Our hypothesis is the second: that speakers SHARE THE LOAD of a difficult utterance in order to EASE PLANNING. It’s less likely that people can pre-plan everything, and we expect to see some degree of BOTH happening instead, and thus we expect to see correlations between latency and duration.

9. Methods Experiment 1: Rosa 2015 PhD dissertation
Experiment 2: Zerkle 2015 First Year Project
Stimuli available at: jaapstimuli.web.unc.edu
Now I will talk about two experiments that were originally designed to examine pronoun usage, but now we want to use to examine acoustic reduction
- The first experiment was part of Elise Rosa’s PhD dissertation this year
- And the second experiment was my own first year project
They are very related, and I will talk about the methods and results together. They are based off of similar paradigms designed by Elise Rosa, and the stimuli is available at this website for anyone to use (and cite)!

10. Experiment 1 Lady Mannerly gave a picnic basket to Sir Barnes.
And then he threw it away.
Lady Mannerly handed the picnic basket to Sir Barnes.
Experiment 1 was an INTERACTIVE Designated Completion task, (again) designed by Elise Rosa.
First the participant was shown an entire murder mystery story in pairs of pictures on a computer screen (53 pairs). The story is novel and engaging, and it uses the same six characters throughout, leading up to a murder of one of the characters at the end.
Example trial: Lady Mannerly gave a picnic basket to Sir Barnes, and then he threw it down the hall.
Then a “detective” confederate came in and told the participant that she needs help solving the crime. The detective described the first picture, and then the participant had to describe the second picture.
CRITICALLY: The participant viewed both pictures for each trial for the entire duration of the trial on his own computer screen.
This design allowed experimental control over the grammatical structure of the first sentence (who is in Subject/Nonsubject positions), and the TYPE of continuation (which character would be mentioned next, goal or source).
Example trial:

11. Experiment 2 Lady Mannerly gave a picnic basket to Sir Barnes.
And then he threw it away.
Sir Barnes received the picnic basket from Lady Mannerly.
Experiment 2 was very similar.
Participant previewed SAME comic story in pairs of pictures on a computer screen (53 pairs)
Detective came in, storyboards with magnetic movable characters/props were displayed between speaker and detective
Detective described first picture and acted it out on the boards (there was also an addressee Gesture manipulation in this experiment, but it is not important in the current analysis (did not have big effects))
CRITICALLY: THEN speaker’s screen displayed the second picture as a prompt
- The speaker described it, and the detective acted it out

12. Methods Experiment 1: Experiment 2:
Only viewed pictures on computer screen
Viewed both pictures for each trial for the entire duration of the trial
Prediction: increases the degree of pre-planning the utterance
Will this have effects on latency and/or duration?
Experiment 2:
Detective enacted on interactive magnet boards
Added audience design factor, made the conversation more engaging
Delayed the point when participants could begin planning the utterance, because they were only shown the second picture after the detective described the first picture
Prediction: increases the degree of incremental planning during the utterance
Here’s a breakdown of the differences between our two experiments

13. Analysis Limited responses: Dependent Measures:
Used a (correct) name description for reference (e.g., Sir Barnes)
Referred to the nonsubject character of the previous sentence (only Experiment 1)
N = 21 (Experiment 1), N = 24 (Experiment 2)
Dependent Measures:
Log of Target Duration = Acoustic Reduction
Log Latency to fluent speech = Planning Time
Likelihood ratings: included as predictors
20 participants indicated which of the two characters was more likely to be mentioned for each item
Goal continuations significantly more likely than source continuations (p=0.0015)
Average rating for each item was used as a predictor
In our analysis, we limited responses to those where the speaker used a (correct) name description for reference.
We did not include trials where pronouns were used to refer to target character, because with these it’s harder to find variation in duration
We also only used trials where the participant Referred to the nonsubject character (because references to subject were usually pronominalized), this limitation is only for Exp1 data. For Exp2, we included both subject and nonsubject continuations, and this didn’t change any of our effects but just increased the amount of items to use in our models.
Target Duration: this is our measure of acoustic reduction (while there are many ways to measure reduction, this is the one we are analyzing here!)
Latency from the beginning of the trial to the participant’s fluent speech: this INCLUDES any potential disfluencies at beginning of utterance
- Again, We’re looking at latency in order to investigate mechanisms of PLANNING TIME
All of our analyses were done using the LOG of these measures, but for ease of interpretation I’ll be using the RAW values in all of my graphs
We also had Likelihood Ratings: A separate group of 20 participants viewed the pictures and indicated which of the two characters was more likely to be mentioned next for each item
From this, we found that Goal continuations were rated as significantly MORE LIKELY than source continuations
And the average rating for each item was used as a predictor

14. Study Goals
Does Thematic Role Predictability affect acoustic reduction?
What is the role of planning?
How are Acoustic Reduction and Planning related?
So back to my Study Goals: The first question we will answer is DOES THEMATIC ROLE PREDICTABILITY AFFECT ACOUSTIC REDUCTION? For this we will look at the Duration measure….

15. Duration Results
GOAL: “Lady Mannerly gave a picnic basket to Sir Barnes. Sir Barnes…”
SOURCE: “The maid took a cookbook from the chef. The chef…”
Are predictable referents (based on Thematic Role) reduced (in terms of duration)?
Are predictable referents reduced?
Up here is a little reminder for what the sentences are like for each continuation, goal and source
DESCRIBE AXES: This graph shows the raw duration of the target name in ms by the type of continuation, goal or source.
In both Experiments, we found NO significant difference in duration between goals and sources.
This is kind of surprising! given the widespread effects in the literature that referential predictability affects acoustic reduction, and the fact that we have found a thematic role effect on pronoun use as well.
So maybe the effects of Thematic Role Predictability aren’t directly on Duration itself, but might be mediated by something like Ease of Planning.
Experiment 1: no significant difference between Goals and Sources
Experiment 2: no significant difference between Goals and Sources

16. Duration Results
GOAL: “Lady Mannerly gave a picnic basket to Sir Barnes. Sir Barnes…”
SOURCE: “The maid took a cookbook from the chef. The chef…”
Another look at our duration results showed an interaction with the Likelihood ratings.
In a previous study, We found that goal continuations are rated as more likely than source continuations.
In the current Experiment 2, we see that The more likely continuations, which happen to be goals, have significantly shorter duration!
EXPLAIN AXES: This graph shows the duration of the target name by the average likelihood ratings for the trials, separated by goal and source continuations.
In other words, goal durations are shorter than sources, when they are more likely.
This likelihood effect is consistent with the ease of Planning idea. Maybe Thematic Role is not the STRONGEST measure of Referential Predictability, but it could be considered as part of a broader concept about how likely or related a certain character or event is in the discourse.
Notably: This effect was not found for experiment 1. We think that this is found only in Experiment 2 because of the potential differences in picture presentation and therefore planning time between the experiments.
Target Likelihood: Goal continuations rated significantly more likely than source continuations (p=0.0015)
Exp2 Goal by Likelihood interaction (p=0.002): Goal durations are shorter than sources when they are more likely

17. Study Goals
Does Thematic Role Predictability affect acoustic reduction?
What is the role of planning?
How are Acoustic Reduction and Planning related?
Next we want to examine the role of PLANNING in terms of latency

18. Latency Results Thematic role affects latency itself
GOAL: “Lady Mannerly gave a picnic basket to Sir Barnes. Sir Barnes…”
SOURCE: “The maid took a cookbook from the chef. The chef…”
Thematic role affects latency itself
First, we want to see if Thematic Role affects Latency itself.
DESCRIBE AXES: This graph shows the latency in ms from the start of the trial to the beginning of fluent speech, by the type of continuation.
In both experiments, Goal continuations were initiated significantly faster than Source continuations.
SO, so far the effects of thematic role seem to be appearing in the latency measure, which means there ARE differences in PLANNING TIME for goals vs sources
We also found that Latencies are significantly longer overall in Experiment 2 (p=0.0001): this again probably has to do with differences in picture presentation between the two experiments. In Experiment 2 the participants didn’t see the next picture until it was their turn to speak, so it makes sense that this lead to longer latencies to begin speaking OVERALL / IN GENERAL.
Experiment 1: Goal continuations initiated faster than source continuations (p=0.01)
Experiment 2: Goal continuations initiated faster than source continuations (p=0.001)
Latencies are significantly longer overall in Experiment 2 (p=0.0001)

19. Study Goals
Does Thematic Role Predictability affect acoustic reduction?
What is the role of planning?
How are Acoustic Reduction and Planning related?
Now we want to see how Duration and Latency are related, and also how thematic role fits in

20. But no significant effect of Thematic Role in either experiment
Latency Results
GOAL: “Lady Mannerly gave a picnic basket to Sir Barnes. Sir Barnes…”
SOURCE: “The maid took a cookbook from the chef. The chef…”
Latency affects duration: shorter latency, shorter duration
EXPLAIN AXES: This graph shows the duration of the target name by the latency to begin speaking.
In both experiments we see a Main effect of latency on duration: if latency to begin speaking is short, target duration is short. If latency to begin speaking is long, target duration is long.
But when we split it by thematic role continuation, there is no significant interaction!
Even though thematic role affects average latency, it does not affect latency when plotted by duration (NOT SURE HOW TO SAY THIS????????)
But no significant effect of Thematic Role in either experiment
Exp1: Target durations were shorter on trials with shorter latencies (p=0.009)
Exp2: Target durations were shorter on trials with shorter latencies (p=0.005)

21. Planning time mediates the effect of thematic role predictability
Speakers initiate utterances more quickly for goals in both experiments
Latency predicts duration in both experiments
No direct effect of thematic role on duration
Except Goal x Target Likelihood interaction in Experiment 2
Thematic Role
Latency
Duration
Thematic Role
Latency
Duration
Thematic Role
Latency
Duration
In summary, we have found that planning time MEDIATES the effect of thematic role predictability on acoustic reduction:
….Say all 3
In Experiment 1, the effect of thematic role on Duration is ENTIRELY MEDIATED by latency
In Experiment 2, the effect of thematic role on Duration is also MEDIATED by latency, but we see an interaction with likelihood that affects duration as well. (Again, if the target character is more likely to be mentioned next (and probably the events are more related), there is shorter target duration. These likely cases happen to be mostly Goal continuations).
This likelihood interaction is consistent with the literature.
On the whole, Thematic Role doesn’t have strong effects on Duration, but taken as a larger measure of likelihood, we do see an effect.
These results show us that the effects of Thematic Role are not independent, which contributes to there being a Planning mechanism, where we also expect to see a strong relationship between latency and duration.
Acoustic Reduction is heavily influenced by variation in the ease of planning an utterance, which in turn is influenced by thematic role predictability.

22. Planning Across the Experiments
Experiment 1 – more pre-planning
Speakers saw both pictures for entire trial
But no direct effect of thematic role on duration, because even sources can be planned out
Experiment 2 – more incremental planning
Speakers only saw second picture when it was their turn to speak, which lead to more communicative pressure to hold the floor
Thematic role predictability effects on duration might only appear when the speaker has to begin speaking quickly but not everything is planned out yet (Exp2), so speakers have to share the load more. Goals will have shorter latency and duration, and sources will have longer latency and duration.
When the speaker has enough time to plan utterance (Exp1), we only see differences in latency (not duration). !
But there’s also the effect of WHEN the speaker can see what they need to talk about that we think ALSO affects utterance PLANNING. This was manipulated in the design of the two experiments.
In Exp1, we see relatively more pre-planning happening. Speakers saw both pictures for the entire trial, and we only found effects of Thematic Role on Latency, NOT Duration, because perhaps even source continuations can be pre-planned out and spoken fluently
In Exp2, we see relatively more incremental planning happening, where speakers SHARE THE LOAD between pre-planning and incremental planning. They only saw the second picture when it was their turn to speak, which lead to more communicative pressure to hold the floor. Here we saw effects of Thematic Role on both Latency and Duration.
This is just speculation, but here’s what I think from our data:
Thematic role predictability effects on duration might only appear when the speaker has to begin speaking quickly but not everything is planned out yet (LIKE IN EXPERIMENT 2), so speakers have to share the load more. Goals will have shorter latency and duration (when rated more likely), and sources will have longer latency and duration (AS WE SAW IN OUR DATA).
When the speaker has enough time to see the pictures and plan their utterance (EXP1), then we only see differences in latency, BUT NOT DURATION, depending on Thematic Role.

23. Conclusions Thematic Role Predictability affects Ease of Planning
Thematic Role predicts Latency
Ease of Planning affects Duration
Strong effects of Latency on Duration
Thematic Role x Likelihood effect in Experiment 2
Stronger effects of predictability when incrementally planning (Experiment 2)
 Referential predictability effects on Duration are mediated by Planning
In conclusion:
We found that Thematic Role Predictability affects the Ease of Planning an utterance. We see this in our data where thematic role predicts latency.
We also found that Ease of Planning affects Duration. We see this in our strong effects of Latency on Duration, and in the Thematic Role by Likelihood effect in Exp2.
We found that there are stronger effects of Predictability on Acoustic Reduction when speakers are INCREMENTALLY Planning (as in Exp2).
This leads us to conclude that Referential Predictability effects on Duration are MEDIATED by Planning
Again, this is fairly speculative!
A lot of things can make it more/less difficult to plan and produce an utterance. Thematic Role is only one form of Referential Predictability,
and maybe it’s not the strongest cause for variation. Maybe we need to investigate other forms of likelihood and Predictability to really figure out how and why planning and production vary!!!
This Study is part of a bigger……

24. Thanks! Dr. Jennifer Arnold Dr. Elise Rosa Kathryn Weatherford
Arnoldlab RAs:
Megan Fullerton
Bryan Smith
Leela Rao
Natasha Vasquez
Taylor Beard
Anita Simha
Grant Huffman
Michaela Neeley
This study is part of a bigger NSF project of Jennifer Arnold’s looking at Predictability, where we’re investigating effects of predictability on both pronoun choice and acoustic variation
I’d like to thank my Advisor at UNC, Jennifer Arnold, for all of her help and guidance and encouragement with this project (and with this talk!).
Thanks to Elise Rosa for starting this project and designing an awesome experiment,
and also thanks to my fellow grad student Kathryn – check out her poster on Prosody and Audience Design at the poster session tomorrow!