2. A brief overview of Speech Recognition and Spoken Language Processing Advanced NLP Guest Lecture August 31 Andrew Rosenberg

3. Speech and NLP Communication in Natural Language Text: –Carefully prepared –Grammatical –Machine readable Typos Sometimes OCR or handwriting issues 1

4. Speech and NLP Communication in Natural Language Speech: –Spontaneous –Less Grammatical –Machine readable with > 10% error using on speech recognition. 2

5. NLP Tasks Parsing Name Tagging Sentiment Analysis Entity Coreference Relation Extraction Machine Translation 3

6. Speech Tasks Parsing –Speech isn’t always grammatical Name Tagging –If a name isn’t “in vocabulary” what do you do? Sentiment Analysis –How the words are spoken helps. Entity Coreference Relation Extraction Machine Translation –how can these handle misrecognition errors? 4

7. Speech Tasks Speech Synthesis Text Normalization Dialog Management Topic Segmentation Language Identification Speaker Identification and Verification –Authorship and security 5

8. The traditional view 6 Text Processing System Named Entity Recognizer Text Processing System Named Entity Recognizer Text Documents Training Application

9. The simplest approach 7 Text Processing System Named Entity Recognizer Text Processing System Named Entity Recognizer Transcribed Documents Text Documents Training Application

10. Speech is errorful text 8 Text Processing System Named Entity Recognizer Text Processing System Named Entity Recognizer Transcribed Documents Training Application

11. Speech signal can be used 9 Text Processing System Named Entity Recognizer Text Processing System Named Entity Recognizer Transcribed Documents Training Application

12. Hybrid speech signal and text 10 Text Processing System Named Entity Recognizer Text Processing System Named Entity Recognizer Transcribed Documents Training Application Text Documents

13. Speech Recognition Standard HMM speech recognition. Front End Acoustic Model Pronunciation Model Language Model Decoding 11

14. Speech Recognition 12 Front End Acoustic Model Pronunciation Model Language Model Word Sequence Acoustic Feature Vector Phone Likelihoods Word Likelihoods

15. Speech Recognition 13 Front End Convert sounds into a sequence of observation vectors Front End Convert sounds into a sequence of observation vectors Language Model Calculate the probability of a sequence of words Language Model Calculate the probability of a sequence of words Pronunciation Model The probability of a pronunciation given a word Pronunciation Model The probability of a pronunciation given a word Acoustic Model The probability of a set of observations given a phone label Acoustic Model The probability of a set of observations given a phone label

16. Front End How do we convert a wave form into a useful representation? We are looking for a vector of numbers which describe the acoustic content Assuming 22kHz 16bit sound. Modeling this directly is not feasible. 14

17. Discrete Cosine Transform Every wave can be decomposed into component sine or cosine waves. Fast Fourier Transform is used to do this efficiently 15

18. Overlapping frames Spectrograms allow for visual inspection of spectral information. We are looking for a compact, numerical representation 16 10ms

19. Single Frame of FFT 17 http://clas.mq.edu.au/acoustics/speech_spectra/fft_lpc_settings.html Australian male /i:/ from “heed” FFT analysis window 12.8ms

20. Example Spectrogram 18

21. “Standard” Representation Mel Frequency Cepstral Coefficients –MFCC 19 Pre- Emphasis window FFT Mel-Filter Bank log FFT -1 Deltas energy 12 MFCC 12 ∆ MFCC 12∆∆ MFCC 1 energy 1 ∆ energy 1 ∆∆ energy

22. Speech Recognition 20 Front End Convert sounds into a sequence of observation vectors Front End Convert sounds into a sequence of observation vectors Language Model Calculate the probability of a sequence of words Language Model Calculate the probability of a sequence of words Pronunciation Model The probability of a pronunciation given a word Pronunciation Model The probability of a pronunciation given a word Acoustic Model The probability of a set of observations given a phone label Acoustic Model The probability of a set of observations given a phone label

23. Language Model What is the probability of a sequence of words? Assume you have a vocabulary of V words. How many possible sequences of N words are there? 21

24. N-gram Language Modeling Simplify the calculation. Big simplifying assumption: Each word is only dependent on the previous N-1 words. 22

25. N-gram Language Modeling Same question. Assume a V word vocabulary, and an N word sequence. How many “counts” are necessary? 23

26. General Language Modeling Any probability calculation can be used here. Class based language models. e.g. Recurrent neural networks 24

27. Speech Recognition 25 Front End Convert sounds into a sequence of observation vectors Front End Convert sounds into a sequence of observation vectors Language Model Calculate the probability of a sequence of words Language Model Calculate the probability of a sequence of words Pronunciation Model The probability of a pronunciation given a word Pronunciation Model The probability of a pronunciation given a word Acoustic Model The probability of a set of observations given a phone label Acoustic Model The probability of a set of observations given a phone label

28. Pronunciation Modeling Identify the likelihood of a phone sequence given a word sequence. There are many simplifying assumptions in pronunciation modeling. 1.The pronunciation of each word is independent of the previous and following. 26

29. Dictionary as Pronunciation Model Assume each word has a single pronunciation 27 IAY CATK AE T THEDH AH HADH AE D ABSURDAH B S ER D YOUY UH D

30. Weighted Dictionary as Pronunciation Model Allow multiple pronunciations and weight each by their likelihood 28 IAY.4 IIH.6 THEDH AH.7 THEDH IY.3 YOUY UH.5 YOUY UW.5

31. Grapheme to Phoneme conversion What about words that you have never seen before? What if you don’t think you’ve seen every possible pronunciation? How do you pronounce: “McKayla”? or “Zoomba”? Try to learn the phonetics of the language. 29

32. Letter to Sound Rules Manually written rules that are able to convert one or more letters to one or more sounds. T -> /t/ H -> /h/ TH -> /dh/ E -> /e/ These rules can get complicated based on the surrounding context. –K is silent when word initial and followed by N. 30

33. Automatic learning of Letter to Sound rules First: Generate an alignment of letters and sounds 31 TEX-T TEHKST TEXT----- ----T KST

34. Automatic learning of Letter to Sound rules Second: Try to learn the mapping automatically. Generate “Features” from the letter sequence Use these feature to predict sounds Almost any machine learning technique can be used. –We’ll use decision trees as an example. 32

35. Decision Trees example Context: L1, L2, p, R1, R2 33 R1 = “h” YesNo Ploophole Fphysics Ftelephone Fgraph Fphoto Ppeanut Ppay Papple øapple øpsycho øpterodactyl øpneumonia Yes No PloopholeFphysics Ftelephone Fgraph Fphoto L1 = “o” R1 = consonant No Yes Ppeanut P pay Papple øpsycho ø pterodactyl øpneumonia

36. Decision Trees example Context: L1, L2, p, R1, R2 34 R1 = “h” YesNo Ploophole Fphysics Ftelephone Fgraph Fphoto Ppeanut Ppay Papple øapple øpsycho øpterodactyl øpneumonia Yes No PloopholeFphysics Ftelephone Fgraph Fphoto L1 = “o” R1 = consonant No Yes Ppeanut P pay Papple øpsycho ø pterodactyl øpneumonia try “PARIS”

37. Decision Trees example Context: L1, L2, p, R1, R2 35 R1 = “h” YesNo Ploophole Fphysics Ftelephone Fgraph Fphoto Ppeanut Ppay Papple øapple øpsycho øpterodactyl øpneumonia Yes No PloopholeFphysics Ftelephone Fgraph Fphoto L1 = “o” R1 = consonant No Yes Ppeanut P pay Papple øpsycho ø pterodactyl øpneumonia Now try “GOPHER”

38. Speech Recognition 36 Language Model Calculate the probability ofa sequence of words Language Model Calculate the probability ofa sequence of words Front End Convert sounds into a sequence of observation vectors Front End Convert sounds into a sequence of observation vectors Language Model Calculate the probability of a sequence of words Language Model Calculate the probability of a sequence of words Pronunciation Model The probability of a pronunciation given a word Pronunciation Model The probability of a pronunciation given a word Acoustic Model The probability of a set of observations given a phone label Acoustic Model The probability of a set of observations given a phone label

39. Acoustic Modeling Hidden markov model. –Used to model the relationship between two sequences. 37

40. Hidden Markov model In a Hidden Markov Model the state sequence is unobserved. Only an observation sequence is available 38 q1q1 q2q2 q3q3 x1x1 x1x1 x2x2 x2x2 x3x3 x3x3

41. Hidden Markov model Observations are MFCC vectors States are phone labels Each state (phone) has an associated GMM modeling the MFCC likelihood 39 q1q1 q2q2 q3q3 x1x1 x1x1 x2x2 x2x2 x3x3 x3x3

42. Training acoustic models TIMIT –close, manual phonetic transcription –2342 sentences Extract MFCC vectors from each frame within each phone For each phone, train a GMM using Expectation Maximization. These GMM is the Acoustic Model. –Common to use 8, or 16 Gaussian Mixture Components. 40

43. Gaussian Mixture Model 41

44. HMM Topology for Training Rather than having one GMM per phone, it is common for acoustic models to represent each phone as 3 triphones 42 S1 S3 S2 S4 S5 /r/

45. 43 Speech in Natural Language Processing ALSO FROM NORTH STATION I THINK THE ORANGE LINE RUNS BY THERE TOO SO YOU CAN ALSO CATCH THE ORANGE LINE AND THEN INSTEAD OF TRANSFERRING UM I YOU KNOW THE MAP IS REALLY OBVIOUS ABOUT THIS BUT INSTEAD OF TRANSFERRING AT PARK STREET YOU CAN TRANSFER AT UH WHAT’S THE STATION NAME DOWNTOWN CROSSING UM AND THAT’LL GET YOU BACK TO THE RED LINE JUST AS EASILY

46. 44 Speech in Natural Language Processing Also, from the North Station... (I think the Orange Line runs by there too so you can also catch the Orange Line... ) And then instead of transferring (um I- you know, the map is really obvious about this but) Instead of transferring at Park Street, you can transfer at (uh what’s the station name) Downtown Crossing and (um) that’ll get you back to the Red Line just as easily.

47. 45 Spoken Language Processing NLP system IR IE QA Summarization Topic Modeling Speech Recognition

48. 46 Spoken Language Processing NLP system IR IE QA Summarization Topic Modeling ALSO FROM NORTH STATION I THINK THE ORANGE LINE RUNS BY THERE TOO SO YOU CAN ALSO CATCH THE ORANGE LINE AND THEN INSTEAD OF TRANSFERRING UM I YOU KNOW THE MAP IS REALLY OBVIOUS ABOUT THIS BUT INSTEAD OF TRANSFERRING AT PARK STREET YOU CAN TRANSFER AT UH WHAT’S THE STATION NAME DOWNTOWN CROSSING UM AND THAT’LL GET YOU BACK TO THE RED LINE JUST AS EASILY

49. 47 Dealing with Speech Errors ALSO FROM NORTH STATION I THINK THE ORANGE LINE RUNS BY THERE TOO SO YOU CAN ALSO CATCH THE ORANGE LINE AND THEN INSTEAD OF TRANSFERRING UM I YOU KNOW THE MAP IS REALLY OBVIOUS ABOUT THIS BUT INSTEAD OF TRANSFERRING AT PARK STREET YOU CAN TRANSFER AT UH WHAT’S THE STATION NAME DOWNTOWN CROSSING UM AND THAT’LL GET YOU BACK TO THE RED LINE JUST AS EASILY Robust NLP system IR IE QA Summarization Topic Modeling

50. 48 Automatic Speech Recognition Assumption ALSO FROM NORTH STATION I THINK THE ORANGE LINE RUNS BY THERE TOO SO YOU CAN ALSO CATCH THE ORANGE LINE AND THEN INSTEAD OF TRANSFERRING UM I YOU KNOW THE MAP IS REALLY OBVIOUS ABOUT THIS BUT INSTEAD OF TRANSFERRING AT PARK STREET YOU CAN TRANSFER AT UH WHAT’S THE STATION NAME DOWNTOWN CROSSING UM AND THAT’LL GET YOU BACK TO THE RED LINE JUST AS EASILY ASR produces a “transcript” of Speech.

51. 49 Automatic Speech Recognition Assumption “Rich Transcription” Also, from the North Station... (I think the Orange Line runs by there too so you can also catch the Orange Line... ) And then instead of transferring (um I- you know, the map is really obvious about this but) Instead of transferring at Park Street, you can transfer at (uh what’s the station name) Downtown Crossing and (um) that’ll get you back to the Red Line just as easily. ASR produces a “transcript” of Speech.

52. 50 Decrease WERIncrease Robustness Speech as Noisy Text Robust NLP system IR IE QA Summarization Topic Modeling Speech Recognition

53. 51 Other directions for improvement. Prosodic Analysis Robust NLP system IR IE QA Summarization Topic Modeling Speech Recognition Use Lattices or N-Best lists

54. Prosody Variation is production properties that lead to changes in intended interpretation. Pitch Intensity Duration, Rhythm, Speaking Rate Spectral Emphasis Pausing 52

55. Tasks that can use prosody Part of Speech Tagging [Eidelman et al. 2010] Parsing [Huang, et al. 2010] Language Modeling [Su & Jelinek, 2008] Pronunciation Modeling [Rosenberg 2012] Acoustic Modeling [Chen, et al. 2006] Emotion Recognition [Lee, et al. 2009] Topic Segmentation [Rosenberg & Hirschberg, 2006, Rosenberg, et al. 2007] Speaker Identification/Verification [Leung, et al. 2008] 53

56. Processing Speech Processing speech is difficult –There are errors in transcripts. –It is not grammatical –The style (genre) of speech is different from the available (text) training data. Processing speech is easy –Speaker information –Intention (sarcasm, certainty, emotion, etc.) –Segmentation 54

57. Questions & Comments What topic was clearest? –murkiest? What was the most interesting? –least interesting? andrew@cs.qc.cuny.edu http://speech.cs.qc.cuny.edu http://eniac.cs.qc.cuny.edu/andrew 55