1. Intonational and Its Meanings
Julia Hirschberg
CS 6998
12/30/2018

2. What do speech researchers do?
Study human production and perception
Try to embody it in machines
Production: TTS, CTS
Perception: ASR, ASRU, speaker ID, language ID
12/30/2018

3. Pitch Accent/Prominence in ToBI
Which items are made intonationally prominent and how?
Accent type:
H* simple high (declarative)
L* simple low (ynq)
L*+H scooped, late rise (uncertainty/ incredulity)
L+H* early rise to stress (contrastive focus)
H+!H* fall onto stress (implied familiarity)
12/30/2018

4. Downstepped accents: !H*, L+!H*, L*+!H Degree of prominence:
within a phrase: HiF0
across phrases
12/30/2018

5. Functions of Pitch Accent
Given/new information
S: Do you need a return ticket.
U: No, thanks, I don’t need a return.
Contrast (narrow focus)
U: No, thanks, I don’t need a RETURN…. (I need a time schedule, receipt,…)
Disambiguation of discourse markers
S: Now let me get you the train information.
U: Okay (thanks) vs. Okay….(but I really want…)
12/30/2018

6. Prosodic Phrasing in ToBI
‘Levels’ of phrasing:
intermediate phrase: one or more pitch accents plus a phrase accent (H or L- )
intonational phrase: 1 or more intermediate phrases + boundary tone (H% or L% )
ToBI break-index tier
0 no word boundary
1 word boundary
2 strong juncture with no tonal markings
3 intermediate phrase boundary
4 intonational phrase boundary
12/30/2018

7. Functions of Phrasing
Disambiguates syntactic constructions, e.g. PP attachment:
S: You should buy the ticket with the discount coupon.
Disambiguates scope ambiguities, e.g. Negation:
S: You aren’t booked through Rome because of the fare.
Or modifier scope:
S: This fare is restricted to retired politicians and civil servants.
12/30/2018

8. L*+H L* H*
H-H%
H-L%
L-H%
L-L%
12/30/2018

9. H* !H*
H+!H*
L+H*
H-H%
H-L%
L-H%
L-L%
12/30/2018

10. Contour Examples
12/30/2018

11. Contours: Accent + Phrasing
What do intonational contours ‘mean’ (Ladd ‘80, Bolinger ‘89)?
Speech acts (statements, questions, requests)
S: That’ll be credit card? (L* H- H%)
Propositional attitude (uncertainty, incredulity)
S: You’d like an evening flight. (L*+H L- H%)
Speaker affect (anger, happiness, love)
U: I said four SEVEN one! (L+H* L- L%)
“Personality”
S: Welcome to the Sunshine Travel System.
12/30/2018

12. Propositional attitude (uncertainty) Did you feed the animals?
I fed the L*+H goldfish L-H%
Distinguish direct/indirect speech acts
Can you open the door?
12/30/2018

13. And Other Things Contribute: Pitch Range and Timing (Rate, Pause)
Level of speaker engagement
Hello vs. HELLO
Contour interpretation
Rise/fall/rise (L*+H L-H%): Elephantiasis isn’t incurable
Discourse/topic structure: paratones
12/30/2018

14. Prosodic Generation for TTS
Corpus-based approaches
Train prosodic variation on large labeled corpora using machine learning techniques
Accent and phrasing decisions
Associate prosodic labels with simple features of transcripts
To do:
Contour variation
TTS default prosodic assignment developed to be independent of domains and tasks. Uses simple text analysis to vary phrasing, accent, possibly pitch range.
While hand-built rule-sets are still used for particular application domains, most systems have moved toward automatically trained prosodic assignment systems.
12/30/2018

15. Timing and backchanneling Disfluencies? Emotion and ‘personality’
Personalized voices
Work in spoken language generation is only beginning as a serious topic of research and development. Along the way there are large questions to answer, both for dialogue and monologue generation:
12/30/2018

16. Concept to Speech Decisions in TTS depend on text analysis
Concept-to-Speech (CTS) systems should be able to do better
System knows what it wants to say and can specify how
But….
Still need labeled corpora to train on
CTS features may be hard to label (focus, given/new,…)
How to decide how to realize these?
In principle, the information TTS systems lack to support natural prosodic assignment is readily available to CTS systems. So the initial hope in the NLG community was that prosodic assignment would be a simple problem. It’s proven however fairly hard. Why?
12/30/2018

17. Prosody in ASRU Little success in improving ASR transcription
More promise in other areas:
Improving rejection
Shrinking search space
Automatic topic segmentation for browsing/retrieval
Identifying ‘salient’ words in turns
Disambiguating speech/dialogue acts: okay
12/30/2018

18. Recognizing communicative ‘problems’
ASR errors
User corrections
‘Aware’ turns
‘Problematic’ dialogues
Disfluencies and self-repairs
Recognizing speaker emotion
12/30/2018

19. Some Research Topics Meaning of intonational contours:
Rise/fall/rise (L*+H L-H%)
A: Did you take out the garbage?
B: Sort of.
A: Sort of!
High rise questions (H* H-H%)
This is the chicken Chermula?
I’m from Skokie?
12/30/2018

20. Compositional theory of intonational meaning (w/Pierrehumbert)
Intonational disambiguation across languages: Spanish, Italian and English (w/Avesani & Prieto)
William isn’t drinking because he’s unhappy
Disfluencies: self-repairs (w/Nakatani)
I want to go to Ba- Baltimore.
Cue phrases (w/Litman)
Now let’s go to work.
Get a3 and a4 for disambig
gw for other
12/30/2018

21. Accent and strict/sloppy interpretations of ellipsis (w/Ward)
People who live in Los Angeles adore it’s beaches and so do people who live in New York
12/30/2018

22. Accent and given/new (w/Terken) The ball touches the circle.
The ball touches the triangle.
The ball touches the cone.
The square touches the ball.
Intonation and discourse structure (w/Grosz & Nakatani)
Boston Directions Corpus
Automatic assignment of accent and phrasing for TTS (w/Wang, Sproat, Koehn, Abney, Collins, Rambow)
12/30/2018

23. ToBI prosodic labeling conventions w/many)
Prosody in dialogue systems (w/Litman & Swerts): generation and understanding (TOOT)
Audio browsing and retrieval: SCAN and SCANMail (w/many)
12/30/2018

24. Potential Projects Build a TTS system in a limited domain
Build a speech recognizer
Study a speech phenomenon (disfluencies, accenting, contours, pitch range variation)
Do some experiments (production, perception).
Examples:
Speech summarization, eye tracking and emotion, deceptive speech, given/new and contour,….
12/30/2018