Predicting Phrasing and Accent Julia Hirschberg CS 6998 12/1/2018

A car bomb attack on a police station in the northern Iraqi city of Kirkuk early Monday killed four civilians and wounded 10 others U.S. military officials said. A leading Shiite member of Iraq's Governing Council on Sunday demanded no more "stalling" on arranging for elections to rule this country once the U.S.-led occupation ends June 30. Abdel Aziz al-Hakim a Shiite cleric and Governing Council member said the U.S.-run coalition should have begun planning for elections months ago. 12/1/2018

Why predict phrasing and accent? TTS and CTS Naturalness Intelligibility Recognition Decrease perplexity Modify durational predictions for words at phrase boundaries Identify most ‘salient’ words 12/1/2018

How predict phrasing and accent? Default prosodic assignment from simple text analysis Hand-built rule-based system: hard to modify and adapt to new domains Corpus-based approaches (Sproat et al ’92) Train prosodic variation on large labeled corpora using machine learning techniques Accent and phrasing decisions TTS default prosodic assignment developed to be independent of domains and tasks. Uses simple text analysis to vary phrasing, accent, possibly pitch range. While hand-built rule-sets are still used for particular application domains, most systems have moved toward automatically trained prosodic assignment systems. 12/1/2018

Associate prosodic labels with simple features of transcripts distance from beginning or end of phrase orthography: punctuation, paragraphing part of speech, constituent information Apply learned rules to new text Note: problems in labeling consistency of e.g. phrase boundaries. How to produce a gold standard?? 12/1/2018

Prosodic Phrasing 2 `levels’ of phrasing in ToBI intermediate phrase: one or more pitch accents plus a phrase accent (H- or L- ) intonational phrase: one or more intermediate phrases + boundary tone (H% or L% ) ToBI break-index tier 0 no word boundary 1 word boundary 2 strong juncture with no tonal markings 3 intermediate phrase boundary 4 intonational phrase boundary 12/1/2018

What are the indicators of phrasing in speech? Timing Pause Lengthening F0 changes Vocal fry/glottalization 12/1/2018

What linguistic and contextual features are linked to phrasing? Syntactic structure? We only suspected they all knew that a burglary had been committed. Abney ’91 chunking Steedman ’90, Oehrle ’91 CCGs … Length of sentence Word co-occurrence statistics I should have known and so should you have. 12/1/2018

Are words on each side accented or not? Where is previous phrase boundary? Part-of-speech of words around potential boundary site Largest constituent dominating w(i) but not w(j) Smallest constituent dominating w(i),w(j) 12/1/2018

Statistical learning methods Classification and regression trees (CART) Rule induction (Ripper) HMMs 12/1/2018

Evaluation: Hard! How to define Gold Standard? Natural speech corpus Multi-speaker/same text Subjective judgments 12/1/2018

Today Incremental improvements continue: Adding higher-accuracy parsing (Koehn et al ‘00) Collins ‘99 parser More sophisticated learning algorithms (Schapire & Singer ‘00) Better representations: tree based? Rules always impoverished Where to next? 12/1/2018

Pitch Accent/Prominence in ToBI Which items are made intonationally prominent and how? Accent type: H* simple high (declarative) L* simple low (ynq) L*+H scooped, late rise (uncertainty/ incredulity) L+H* early rise to stress (contrastive focus) H+!H* fall onto stress (implied familiarity) 12/1/2018

What are the indicators of accent? F0 excursion Durational lengthening Voice quality Vowel quality Loudness 12/1/2018

What are the correlates of accent in text and context? Word class: content vs. function words Information status: Given/new Topic/Focus Contrast Grammatical function The ball touches the cone. Surface position: Today George is hungry. 12/1/2018

Association with focus: Dogs must be carried. John only introduced Mary to Sue. 12/1/2018

How can we capture such information simply? POS window Position of candidate word in sentence Location of prior phrase boundary Pseudo-given/new Location of word in complex nominal and stress prediction for that nominal Word co-occurrence 12/1/2018

Today Concept-to-Speech (CTS) – Pan&McKeown99 systems should be able to specify “better” prosody: the system knows what it wants to say and can specify how New features! New learning methods: Boosting and bagging (Sun02) Combine text and acoustic cues for ASR Co-training?? In principle, the information TTS systems lack to support natural prosodic assignment is readily available to CTS systems. So the initial hope in the NLG community was that prosodic assignment would be a simple problem. It’s proven however fairly hard. Why? 12/1/2018

MAGIC MM system for presenting cardiac patient data Developed at Columbia by McKeown and colleagues in conjunction with Columbia Presbyterian Medical Center to automate post-operative status reporting for bypass patients Uses mostly traditional NLG hand-developed components Generate text, then annotate prosodically Corpus-trained prosodic assignment component To give an example of how more traditional NLG systems are coping with these problems, I want to briefly describe some work Shimei Pan has done for her doctoral thesis at Columbia University with Kathy McKeown and me on training a prosody module for a medical domain. 12/1/2018

Corpus: written and oral patient reports 50min multi-speaker, spontaneous + 11min single speaker, read 1.24M word text corpus of discharge summaries Transcribed, ToBI labeled Generator features labeled/extracted: syntactic function 12/1/2018

semantic ‘informativeness’ (rarity in corpus) p.o.s. semantic category semantic ‘informativeness’ (rarity in corpus) semantic constituent boundary location and length salience given/new focus theme/ rheme ‘importance’ ‘unexpectedness’ Lessons I personally learned from this: First, just because CTS systems have more information doesn’t mean this information will be useful for prosodic assignment. Second, there is something unsatisfactory about viewing prosodic decisions being made after the text has been constructed. 12/1/2018

Very hard to label features Results: new features to specify TTS prosody Of CTS-specific features only semantic informativeness (likeliness of occurring in a corpus) useful so far Looking at context, word collocation for accent placement helps predict accent RED CELL (less predictable) vs. BLOOD cell (more) Most predictable words are accented less frequently (40-46%) and least predictable more (73-80%) Unigram+bigram model predicts accent status w/77% (+/-.51) accuracy 12/1/2018

Future Intonation Prediction Assigning affect (emotion) Personalizing TTS Conveying personality, charisma? 12/1/2018

Next Week Look at 2 other phenomena we’d like to capture in TTS systems: Information status Discourse (topic) structure Homework 2 is due on Monday! 12/1/2018