Download presentation
Presentation is loading. Please wait.
1
Speech Processing Speech Recognition
August 12, 2005 11/17/2018
2
Speech Recognition Applications of Speech Recognition (ASR) Dictation
Telephone-based Information (directions, air travel, banking, etc) Hands-free (in car) Speaker Identification Language Identification Second language ('L2') (accent reduction) Audio archive searching 11/17/2018
3
LVCSR Large Vocabulary Continuous Speech Recognition
~20,000-64,000 words Speaker independent (vs. speaker-dependent) Continuous speech (vs isolated-word) 11/17/2018
4
LVCSR Design Intuition
Build a statistical model of the speech-to-words process Collect lots and lots of speech, and transcribe all the words. Train the model on the labeled speech Paradigm: Supervised Machine Learning + Search 11/17/2018
5
Speech Recognition Architecture
Speech Waveform Spectral Feature Vectors Phone Likelihoods P(o|q) Words 1. Feature Extraction (Signal Processing) 2. Acoustic Model Phone Likelihood Estimation (Gaussians or Neural Networks) 5. Decoder (Viterbi or Stack Decoder) 4. Language Model (N-gram Grammar) 3. HMM Lexicon 11/17/2018
6
The Noisy Channel Model
Search through space of all possible sentences. Pick the one that is most probable given the waveform. 11/17/2018
7
The Noisy Channel Model (II)
What is the most likely sentence out of all sentences in the language L given some acoustic input O? Treat acoustic input O as sequence of individual observations O = o1,o2,o3,…,ot Define a sentence as a sequence of words: W = w1,w2,w3,…,wn 11/17/2018
8
Noisy Channel Model (III)
Probabilistic implication: Pick the highest prob S: We can use Bayes rule to rewrite this: Since denominator is the same for each candidate sentence W, we can ignore it for the argmax: 11/17/2018
9
A quick derivation of Bayes Rule
Conditionals Rearranging And also 11/17/2018
10
Bayes (II) We know… So rearranging things 11/17/2018
11
Noisy channel model likelihood prior 11/17/2018
12
The noisy channel model
Ignoring the denominator leaves us with two factors: P(Source) and P(Signal|Source) 11/17/2018
13
Speech Architecture meets Noisy Channel
11/17/2018
14
Five easy pieces Feature extraction Acoustic Modeling
HMMs, Lexicons, and Pronunciation Decoding Language Modeling 11/17/2018
15
Feature Extraction Digitize Speech Extract Frames 11/17/2018
16
Digitizing Speech 11/17/2018
17
Digitizing Speech (A-D)
Sampling: measuring amplitude of signal at time t 16,000 Hz (samples/sec) Microphone (“Wideband”): 8,000 Hz (samples/sec) Telephone Why? Need at least 2 samples per cycle max measurable frequency is half sampling rate Human speech < 10,000 Hz, so need max 20K Telephone filtered at 4K, so 8K is enough 11/17/2018
18
Digitizing Speech (II)
Quantization Representing real value of each amplitude as integer 8-bit (-128 to 127) or 16-bit ( to 32767) Formats: 16 bit PCM 8 bit mu-law; log compression LSB (Intel) vs. MSB (Sun, Apple) Headers: Raw (no header) Microsoft wav Sun .au 40 byte header 11/17/2018
19
Frame Extraction . . . A frame (25 ms wide) extracted every 10 ms
a a a3 Figure from Simon Arnfield 11/17/2018
20
MFCC (Mel Frequency Cepstral Coefficients)
Do FFT to get spectral information Like the spectrogram/spectrum we saw earlier Apply Mel scaling Linear below 1kHz, log above, equal samples above and below 1kHz Models human ear; more sensitivity in lower freqs Plus Discrete Cosine Transformation 11/17/2018
21
Final Feature Vector 39 Features per 10 ms frame:
12 MFCC features 12 Delta MFCC features 12 Delta-Delta MFCC features 1 (log) frame energy 1 Delta (log) frame energy 1 Delta-Delta (log frame energy) So each frame represented by a 39D vector 11/17/2018
22
Where we are Given: a sequence of acoustic feature vectors, one every 10 ms Goal: output a string of words We’ll spend 6 lectures on how to do this Rest of today: Markov Models Hidden Markov Models in the abstract Forward Algorithm Viterbi Algorithm Start of HMMs for speech 11/17/2018
23
Acoustic Modeling Given a 39d vector corresponding to the observation of one frame oi And given a phone q we want to detect Compute p(oi|q) Most popular method: GMM (Gaussian mixture models) Other methods MLP (multi-layer perceptron) 11/17/2018
24
Acoustic Modeling: MLP computes p(q|o)
11/17/2018
25
Gaussian Mixture Models
Also called “fully-continuous HMMs” P(o|q) computed by a Gaussian: 11/17/2018
26
Gaussians for Acoustic Modeling
A Gaussian is parameterized by a mean and a variance: Different means P(o|q): P(o|q) is highest here at mean P(o|q is low here, very far from mean) P(o|q) o 11/17/2018
27
Training Gaussians A (single) Gaussian is characterized by a mean and a variance Imagine that we had some training data in which each phone was labeled We could just compute the mean and variance from the data: 11/17/2018
28
But we need 39 gaussians, not 1!
The observation o is really a vector of length 39 So need a vector of Gaussians: 11/17/2018
29
Actually, mixture of gaussians
Each phone is modeled by a sum of different gaussians Hence able to model complex facts about the data Phone A Phone B 11/17/2018
30
Gaussians acoustic modeling
Summary: each phone is represented by a GMM parameterized by M mixture weights M mean vectors M covariance matrices Usually assume covariance matrix is diagonal I.e. just keep separate variance for each cepstral feature 11/17/2018
31
ASR Lexicon: Markov Models for pronunciation
11/17/2018
32
The Hidden Markov model
11/17/2018
33
Formal definition of HMM
States: a set of states Q = q1, q2…qN Transition probabilities: a set of probabilities A = a01,a02,…an1,…ann. Each aij represents P(j|i) Observation likelihoods: a set of likelihoods B=bi(ot), probability that state i generated observation t Special non-emitting initial and final states 11/17/2018
34
Pieces of the HMM Observation likelihoods (‘b’), p(o|q), represents the acoustics of each phone, and are computed by the gaussians (“Acoustic Model”, or AM) Transition probabilities represent the probability of different pronunciations (different sequences of phones) States correspond to phones 11/17/2018
35
Pieces of the HMM Actually, I lied when I say states correspond to phones Actually states usually correspond to triphones CHEESE (phones): ch iy z CHEESE (triphones) #-ch+iy, ch-iy+z, iy-z+# 11/17/2018
36
Pieces of the HMM Actually, I lied again when I said states correspond to triphones In fact, each triphone has 3 states for beginning, middle, and end of the triphone. 11/17/2018
37
A real HMM 11/17/2018
38
Cross-word triphones Word-Internal Context-Dependent Models
‘OUR LIST’: SIL AA+R AA-R L+IH L-IH+S IH-S+T S-T Cross-Word Context-Dependent Models SIL-AA+R AA-R+L R-L+IH L-IH+S IH-S+T S-T+SIL 11/17/2018
39
Summary ASR Architecture Five easy pieces of an ASR system
The Noisy Channel Model Five easy pieces of an ASR system Feature Extraction: 39 “MFCC” features Acoustic Model: Gaussians for computing p(o|q) Lexicon/Pronunciation Model HMM: Next time: Decoding: how to combine these to compute words from speech! 11/17/2018
40
Perceptual properties
Pitch: perceptual correlate of frequency Loudness: perceptual correlate of power, which is related to square of amplitude 11/17/2018
41
Speech Recognition Applications of Speech Recognition (ASR) Dictation
Telephone-based Information (directions, air travel, banking, etc) Hands-free (in car) Speaker Identification Language Identification Second language ('L2') (accent reduction) Audio archive searching 11/17/2018
42
LVCSR Large Vocabulary Continuous Speech Recognition
~20,000-64,000 words Speaker independent (vs. speaker-dependent) Continuous speech (vs isolated-word) 11/17/2018
43
LVCSR Design Intuition
Build a statistical model of the speech-to-words process Collect lots and lots of speech, and transcribe all the words. Train the model on the labeled speech Paradigm: Supervised Machine Learning + Search 11/17/2018
44
Speech Recognition Architecture
Speech Waveform Spectral Feature Vectors Phone Likelihoods P(o|q) Words 1. Feature Extraction (Signal Processing) 2. Acoustic Model Phone Likelihood Estimation (Gaussians or Neural Networks) 5. Decoder (Viterbi or Stack Decoder) 4. Language Model (N-gram Grammar) 3. HMM Lexicon 11/17/2018
45
The Noisy Channel Model
Search through space of all possible sentences. Pick the one that is most probable given the waveform. 11/17/2018
46
The Noisy Channel Model (II)
What is the most likely sentence out of all sentences in the language L given some acoustic input O? Treat acoustic input O as sequence of individual observations O = o1,o2,o3,…,ot Define a sentence as a sequence of words: W = w1,w2,w3,…,wn 11/17/2018
47
Noisy Channel Model (III)
Probabilistic implication: Pick the highest prob S: We can use Bayes rule to rewrite this: Since denominator is the same for each candidate sentence W, we can ignore it for the argmax: 11/17/2018
48
A quick derivation of Bayes Rule
Conditionals Rearranging And also 11/17/2018
49
Bayes (II) We know… So rearranging things 11/17/2018
50
Noisy channel model likelihood prior 11/17/2018
51
The noisy channel model
Ignoring the denominator leaves us with two factors: P(Source) and P(Signal|Source) 11/17/2018
52
Five easy pieces Feature extraction Acoustic Modeling
HMMs, Lexicons, and Pronunciation Decoding Language Modeling 11/17/2018
53
Feature Extraction Digitize Speech Extract Frames 11/17/2018
54
Digitizing Speech 11/17/2018
55
Digitizing Speech (A-D)
Sampling: measuring amplitude of signal at time t 16,000 Hz (samples/sec) Microphone (“Wideband”): 8,000 Hz (samples/sec) Telephone Why? Need at least 2 samples per cycle max measurable frequency is half sampling rate Human speech < 10,000 Hz, so need max 20K Telephone filtered at 4K, so 8K is enough 11/17/2018
56
Digitizing Speech (II)
Quantization Representing real value of each amplitude as integer 8-bit (-128 to 127) or 16-bit ( to 32767) Formats: 16 bit PCM 8 bit mu-law; log compression LSB (Intel) vs. MSB (Sun, Apple) Headers: Raw (no header) Microsoft wav Sun .au 40 byte header 11/17/2018
57
Frame Extraction . . . A frame (25 ms wide) extracted every 10 ms
a a a3 Figure from Simon Arnfield 11/17/2018
58
MFCC (Mel Frequency Cepstral Coefficients)
Do FFT to get spectral information Like the spectrogram/spectrum we saw earlier Apply Mel scaling Linear below 1kHz, log above, equal samples above and below 1kHz Models human ear; more sensitivity in lower freqs Plus Discrete Cosine Transformation 11/17/2018
59
Final Feature Vector 39 Features per 10 ms frame:
12 MFCC features 12 Delta MFCC features 12 Delta-Delta MFCC features 1 (log) frame energy 1 Delta (log) frame energy 1 Delta-Delta (log frame energy) So each frame represented by a 39D vector 11/17/2018
60
Acoustic Modeling Given a 39d vector corresponding to the observation of one frame oi And given a phone q we want to detect Compute p(oi|q) Most popular method: GMM (Gaussian mixture models) Other methods MLP (multi-layer perceptron) 11/17/2018
61
Acoustic Modeling: MLP computes p(q|o)
11/17/2018
62
Gaussian Mixture Models
Also called “fully-continuous HMMs” P(o|q) computed by a Gaussian: 11/17/2018
63
Gaussians for Acoustic Modeling
A Gaussian is parameterized by a mean and a variance: Different means P(o|q): P(o|q) is highest here at mean P(o|q is low here, very far from mean) P(o|q) o 11/17/2018
64
Training Gaussians A (single) Gaussian is characterized by a mean and a variance Imagine that we had some training data in which each phone was labeled We could just compute the mean and variance from the data: 11/17/2018
65
But we need 39 gaussians, not 1!
The observation o is really a vector of length 39 So need a vector of Gaussians: 11/17/2018
66
Actually, mixture of gaussians
Each phone is modeled by a sum of different gaussians Hence able to model complex facts about the data Phone A Phone B 11/17/2018
67
Gaussians acoustic modeling
Summary: each phone is represented by a GMM parameterized by M mixture weights M mean vectors M covariance matrices Usually assume covariance matrix is diagonal I.e. just keep separate variance for each cepstral feature 11/17/2018
68
ASR Lexicon: Markov Models for pronunciation
11/17/2018
69
The Hidden Markov model
11/17/2018
70
Formal definition of HMM
States: a set of states Q = q1, q2…qN Transition probabilities: a set of probabilities A = a01,a02,…an1,…ann. Each aij represents P(j|i) Observation likelihoods: a set of likelihoods B=bi(ot), probability that state i generated observation t Special non-emitting initial and final states 11/17/2018
71
Pieces of the HMM Observation likelihoods (‘b’), p(o|q), represents the acoustics of each phone, and are computed by the gaussians (“Acoustic Model”, or AM) Transition probabilities represent the probability of different pronunciations (different sequences of phones) States correspond to phones 11/17/2018
72
Pieces of the HMM Actually, I lied when I say states correspond to phones Actually states usually correspond to triphones CHEESE (phones): ch iy z CHEESE (triphones) #-ch+iy, ch-iy+z, iy-z+# 11/17/2018
73
Pieces of the HMM Actually, I lied again when I said states correspond to triphones In fact, each triphone has 3 states for beginning, middle, and end of the triphone. 11/17/2018
74
A real HMM 11/17/2018
75
Cross-word triphones Word-Internal Context-Dependent Models
‘OUR LIST’: SIL AA+R AA-R L+IH L-IH+S IH-S+T S-T Cross-Word Context-Dependent Models SIL-AA+R AA-R+L R-L+IH L-IH+S IH-S+T S-T+SIL 11/17/2018
76
Summary ASR Architecture Five easy pieces of an ASR system
The Noisy Channel Model Five easy pieces of an ASR system Feature Extraction: 39 “MFCC” features Acoustic Model: Gaussians for computing p(o|q) Lexicon/Pronunciation Model HMM: Next time: Decoding: how to combine these to compute words from speech! 11/17/2018
Similar presentations
© 2024 SlidePlayer.com. Inc.
All rights reserved.