Pronunciation Modeling Lecture 11 Spoken Language Processing Prof. Andrew Rosenberg

What is a pronunciation model? 1 Acoustic Model Acoustic Model Pronunciation Model Pronunciation Model Language Model Language Model Audio Features Phone Hypothese Word Hypothese

Why do we need one? The pronunciation model defines the mapping between sequences of phones and words. The acoustic model can deliver a one- best, hypothesis – “best guess”. From this single guess, converting to words can be done with dynamic programming alignment. Or viewed as a Finite State Automata. 2

Simplest Pronunciation “model” A dictionary. Associate a word (lexical item, orthographic form) with a pronunciation. 3 ACHEEY K ACHESEY K S ADJUNCTAE JH AH NG K T ADJUNCTSAE JH AN NG K T S ADVANTAGEAH D V AE N T IH JH ADVANTAGEAH D V AE N IH JH ADVANTAGEAH D V AE N T AH JH

Example of a pronunciation dictionary 4

Finite State Automata view Each word is an automata over phones 5 EY K K K K AH D D V V AE N N T T S S I I JH

Size of whole word models these models get very big, very quickly 6 EY K K K K AH D D V V AE N N T T S S I I JH START END

Potential problems Every word in the training material and test vocabulary must be in the dictionary The dictionary is generally written by hand Prone to errors and inconsistencies 7 ACHEEY K ACHESEY K S ADJUNCTAE JH AH NG K T ADJUNCTSAE JH AN NG K T S ADVANTAGEAH D V AE N T IH JH ADVANTAGEAH D V AE N IH JH ADVANTAGEAH D V AE N T AH JH

Baseforms represented by graphs 8

Composition From the word graph, we can replace each phone by its markov model 9

Automating the construction Do we need to write a rule for every word? pluralizing? –Where is it +[Z]? +[IH Z]? prefixes, unhappy, etc. –+[UH N] –How can you tell the difference between “unhappy”, “unintelligent” and “under” and “ 10

Is every pronunciation equally likely? Different phonetic realizations can be weighted. The FSA view of the pronunciation model makes this easy. 11 ACAPULCOAE K AX P AH L K OW ACAPULCOAA K AX P UH K OW THETH IY THE TH AX PROBABLYP R AA B AX B L IY PROBABLYP R AA B L IY PROBABLYP R AA L IY

Is every pronunciation equally likely? Different phonetic realizations can be weighted. The FSA view of the pronunciation model makes this easy. 12 ACAPULCOAE K AX P AH L K OW0.75 ACAPULCOAA K AX P UH K OW0.25 THETH IY0.15 THE TH AX0.85 PROBABLYP R AA B AX B L IY0.5 PROBABLYP R AA B L IY0.4 PROBABLYP R AA L IY0.1

Collecting pronunciations Collect a lot of data Ask a phonetician to phonetically transcribe the data. Count how many times each production is observed. This is very expensive – time consuming, finding linguists. 13

Collecting pronunciations Start with equal likelihoods of all pronunciations Run the recognizer on transcribed speech –forced alignment See how many times the recognizer uses each pronunciation. Much cheaper, but less reliable 14

Out of Vocabulary Words A major problem for Dictionary based pronunciation is out of vocabulary terms. If you’ve never seen a name, or new word, how do you know how to pronounce it? –Person names –Organization and Company Names –New words “truthiness”, “hypermiling”, “woot”, “app” –Medical, scientific and technical terms 15

Collecting Pronunciations from the web Newspapers, blog posts etc. often use new names and unknown terms. For example: –Flickeur (pronounced like Voyeur) randomly retrieves images from Flickr.com and creates an infinite film with a style that can vary between stream-of-consciousness, documentary or video clip. –Our group traveled to Peterborough (pronounced like “Pita-borough”)... The web can be mined for pronunciations [Riley, Jansche, Ramabhadran 2009] 16

Grapheme to Phoneme Conversion Given a new word, how do you pronounce it. Grapheme is a language independent term for things like “letters”, “characters”, “kanji”, etc. With a phoneme to grapheme-to-phoneme converter, dictionaries can be augmented with any word. Some languages are more ambiguous than others. 17

Grapheme to Phoneme conversion Goal: Learn an alignment between graphemes (letters) and phonemes (sounds) Find the lowest cost alignment. Weight rules, and learn contextual variants. 18 TEX-T TEHKST TEXT T KST

Grapheme to Phoneme Difficulties How to deal with Abbreviations –US CENSUS –NASA, scuba vs. AT&T, ASR –LOL –IEEE What about misspellings? –should “teh” have an entry in the dictionary? –If we’re collecting new terms from the web, or other unreliable sources, how do we know what is a new word? 19

Application of Grapheme to Phoneme Conversion This Pronunciation Model is used much more often in Speech Synthesis than Speech Recognition In Speech Recognition we’re trying to do Phoneme-to-Grapheme conversion –This is a very tricky problem. –“ghoti” -> F IH SH –“ghoti” -> silence 20

Approaches to Grapheme to Phoneme conversion “Instance Based Learning” –Lookup based on a sliding window of 3 letters –Helps with sounds like “ch” and “sh” Hidden Markov Model –Observations are phones –States are letters 21

Machine Learning for Grapheme to Phoneme Conversion Input: –A letter, and surrounding context, e.g. 2 previous and 2 following letters Output: –Phoneme 22

Decision Trees Decision trees are intuitive classifiers –Classifier: supervised machine learning, generating categorical predictions 23 Feature > threshold? Class A Class B

Decision Trees Example 24

Decision Tree Training How does the letter “p” sound? Training data –Ploophole, peanuts, pay, apple –Fphysics, telephone, graph, photo –øapple, psycho, pterodactyl, pneumonia pronunciation depends on context 25

Decision Trees example Context: L1, L2, p, R1, R2 26 R1 = “h” YesNo Ploophole Fphysics Ftelephone Fgraph Fphoto Ppeanut Ppay Papple øapple øpsycho øpterodactyl øpneumonia

Decision Trees example Context: L1, L2, p, R1, R2 27 R1 = “h” YesNo Ploophole Fphysics Ftelephone Fgraph Fphoto Ppeanut Ppay Papple øapple øpsycho øpterodactyl øpneumonia Yes No PloopholeFphysics Ftelephone Fgraph Fphoto L1 = “o” R1 = consonant No Yes Ppeanut P pay Papple øpsycho ø pterodactyl øpneumonia

Decision Trees example Context: L1, L2, p, R1, R2 28 R1 = “h” YesNo Ploophole Fphysics Ftelephone Fgraph Fphoto Ppeanut Ppay Papple øapple øpsycho øpterodactyl øpneumonia Yes No PloopholeFphysics Ftelephone Fgraph Fphoto L1 = “o” R1 = consonant No Yes Ppeanut P pay Papple øpsycho ø pterodactyl øpneumonia try “PARIS”

Decision Trees example Context: L1, L2, p, R1, R2 29 R1 = “h” YesNo Ploophole Fphysics Ftelephone Fgraph Fphoto Ppeanut Ppay Papple øapple øpsycho øpterodactyl øpneumonia Yes No PloopholeFphysics Ftelephone Fgraph Fphoto L1 = “o” R1 = consonant No Yes Ppeanut P pay Papple øpsycho ø pterodactyl øpneumonia Now try “GOPHER”

Training a Decision Tree At each node, decide what the most useful split is. –Consider all features –Select the one that improves the performance the most There are a few ways to calculate improved performance –Information Gain is typically used. –Accuracy is less common. Can require many evaluations 30

Pronunciation Models in TTS and ASR In ASR, we have phone hypotheses from the acoustic model, and need word hypotheses. In TTS, we have the desired word, but need a corresponding phone sequence to synthesize. 31

Next Class Language Modeling Reading: J&M Chapter 4 32