1. Statistical NLP Spring 2010
Lecture 8: Speech Signal
Dan Klein – UC Berkeley

2. Speech in a Slide s p ee ch l a b
Frequency gives pitch; amplitude gives volume
Frequencies at each time slice processed into observation vectors
s p ee ch l a b
amplitude
frequency
……………………………………………..a12a13a12a14a14………..

3. Articulatory System Nasal cavity Oral cavity Pharynx
Vocal folds (in the larynx)
Trachea
Lungs
Oral cavity
Sagittal section of the vocal tract (Techmer 1880)
Text from Ohala, Sept 2001, from Sharon Rose slide

4. Places of Articulation
alveolar
post-alveolar/palatal
dental
velar
uvular
labial
pharyngeal
laryngeal/glottal
Figure thanks to Jennifer Venditti

5. Labial place Bilabial: p, b, m labiodental Labiodental: f, v bilabial
Figure thanks to Jennifer Venditti

6. Coronal place alveolar post-alveolar/palatal dental Dental: th/dh
t/d/s/z/l/n
Post:
sh/zh/y
Figure thanks to Jennifer Venditti

7. Dorsal Place Velar: k/g/ng velar uvular pharyngeal
Figure thanks to Jennifer Venditti

8. Space of Phonemes
Standard international phonetic alphabet (IPA) chart of consonants

9. Manner of Articulation
In addition to varying by place, sounds vary by manner
Stop: complete closure of articulators, no air escapes via mouth
Oral stop: palate is raised (p, t, k, b, d, g)
Nasal stop: oral closure, but palate is lowered (m, n, ng)
Fricatives: substantial closure, turbulent: (f, v, s, z)
Approximants: slight closure, sonorant:
(l, r, w)
Vowels: no closure, sonorant: (i, e, a)

10. Space of Phonemes
Standard international phonetic alphabet (IPA) chart of consonants

11. Oral vs. Nasal Sounds
Figure from Jong-bok Kim

12. Vowels IY AA UW
Figure from Eric Keller

13. Vowel Space

14. “She just had a baby” What can we learn from a wavefile?
No gaps between words (!)
Vowels are voiced, long, loud
Length in time = length in space in waveform picture
Voicing: regular peaks in amplitude
When stops closed: no peaks, silence
Peaks = voicing: .46 to .58 (vowel [iy], from second .65 to .74 (vowel [ax]) and so on
Silence of stop closure (1.06 to 1.08 for first [b], or 1.26 to 1.28 for second [b])
Fricatives like [sh]: intense irregular pattern; see .33 to .46

15. Non-Local Cues
pat
pad
bad
spat
Example from Ladefoged

16. Simple Periodic Waves of Sound
Y axis: Amplitude = amount of air pressure at that point in time
Zero is normal air pressure, negative is rarefaction
X axis: Time.
Frequency = number of cycles per second.
20 cycles in .02 seconds = 1000 cycles/second = 1000 Hz

17. Complex Waves: 100Hz+1000Hz

18. Spectrum Frequency components (100 and 1000 Hz) on x-axis Amplitude
Frequency in Hz

19. Spectrum of an Actual Soundwave

20. Waveforms for Speech Waveform of the vowel [iy]
Frequency: repetitions/second of a wave
Above vowel has 28 reps in .11 secs
So freq is 28/.11 = 255 Hz
This is speed that vocal folds move, hence voicing
Amplitude: y axis: amount of air pressure at that point in time
Zero is normal air pressure, negative is rarefaction

21. Part of [ae] waveform from “had”
Note complex wave repeating nine times in figure
Plus smaller waves which repeats 4 times for every large pattern
Large wave has frequency of 250 Hz (9 times in .036 seconds)
Small wave roughly 4 times this, or roughly 1000 Hz
Two little tiny waves on top of peak of 1000 Hz waves

22. Back to Spectra Spectrum represents these freq components
Computed by Fourier transform, algorithm which separates out each frequency component of wave.
x-axis shows frequency, y-axis shows magnitude (in decibels, a log measure of amplitude)
Peaks at 930 Hz, 1860 Hz, and 3020 Hz.

23. Why these Peaks? Articulator process:
The vocal cord vibrations create harmonics
The mouth is an amplifier
Depending on shape of mouth, some harmonics are amplified more than others

24. Vowel [i] sung at successively higher pitches
F#2 A2 C3
F#3 A3
C4 (middle C) A4
Figures from Ratree Wayland

25. Deriving Schwa Reminder of basic facts about sound waves f = c/
c = speed of sound (approx 35,000 cm/sec)
A sound with =10 meters: f = 35 Hz (35,000/1000)
A sound with =2 centimeters: f = 17,500 Hz (35,000/2)

26. Resonances of the Vocal Tract
The human vocal tract as an open tube:
Air in a tube of a given length will tend to vibrate at resonance frequency of tube.
Constraint: Pressure differential should be maximal at (closed) glottal end and minimal at (open) lip end.
Closed end
Open end
Length 17.5 cm.
Figure from W. Barry

27. From Sundberg

28. Computing the 3 Formants of Schwa
Let the length of the tube be L
F1 = c/1 = c/(4L) = 35,000/4*17.5 = 500Hz
F2 = c/2 = c/(4/3L) = 3c/4L = 3*35,000/4*17.5 = 1500Hz
F3 = c/3 = c/(4/5L) = 5c/4L = 5*35,000/4*17.5 = 2500Hz
So we expect a neutral vowel to have 3 resonances at 500, 1500, and 2500 Hz
These vowel resonances are called formants

29. From
Mark
Liberman’s
Web site

30. Seeing Formants: the Spectrogram

31. Vowel Space

32. American English Vowel Space
FRONT
BACK
HIGH
LOW iy ih eh ae aa ao uw uh ah ax ix ux ey ow aw oy ay
Figures from Jennifer Venditti, H. T. Bunnell

33. Dialect Issues
American
British
Speech varies from dialect to dialect (examples are American vs. British English)
Syntactic (“I could” vs. “I could do”)
Lexical (“elevator” vs. “lift”)
Phonological
Phonetic
Mismatch between training and testing dialects can cause a large increase in error rate
all
old

34. How to Read Spectrograms
bab: closure of lips lowers all formants: so rapid increase in all formants at beginning of "bab”
dad: first formant increases, but F2 and F3 slight fall
gag: F2 and F3 come together: this is a characteristic of velars. Formant transitions take longer in velars than in alveolars or labials
From Ladefoged “A Course in Phonetics”

35. “She came back and started again”
1. lots of high-freq energy
3. closure for k
4. burst of aspiration for k
5. ey vowel; faint 1100 Hz formant is nasalization
6. bilabial nasal
7. short b closure, voicing barely visible.
8. ae; note upward transitions after bilabial stop at beginning
9. note F2 and F3 coming together for "k"
From Ladefoged “A Course in Phonetics”

37. The Noisy Channel Model
Search through space of all possible sentences.
Pick the one that is most probable given the waveform.

38. Speech Recognition Architecture

39. Digitizing Speech

40. Frame Extraction . . . A frame (25 ms wide) extracted every 10 ms
a a a3
Figure from Simon Arnfield

41. Mel Freq. Cepstral Coefficients
Do FFT to get spectral information
Like the spectrogram/spectrum we saw earlier
Apply Mel scaling
Linear below 1kHz, log above, equal samples above and below 1kHz
Models human ear; more sensitivity in lower freqs
Plus Discrete Cosine Transformation

42. Final Feature Vector 39 (real) features per 10 ms frame:
12 MFCC features
12 Delta MFCC features
12 Delta-Delta MFCC features
1 (log) frame energy
1 Delta (log) frame energy
1 Delta-Delta (log frame energy)
So each frame is represented by a 39D vector

43. HMMs for Speech

44. Phones Aren’t Homogeneous

45. Need to Use Subphones

46. A Word with Subphones

47. Viterbi Decoding

48. ASR Lexicon: Markov Models

49. HMMs for Continuous Observations?
Before: discrete, finite set of observations
Now: spectral feature vectors are real-valued!
Solution 1: discretization
Solution 2: continuous emissions models
Gaussians
Multivariate Gaussians
Mixtures of Multivariate Gaussians
A state is progressively:
Context independent subphone (~3 per phone)
Context dependent phone (=triphones)
State-tying of CD phone

50. Vector Quantization Idea: discretization
Map MFCC vectors onto discrete symbols
Compute probabilities just by counting
This is called Vector Quantization or VQ
Not used for ASR any more; too simple
Useful to consider as a starting point

51. Gaussian Emissions VQ is insufficient for real ASR
Instead: Assume the possible values of the observation vectors are normally distributed.
Represent the observation likelihood function as a Gaussian with mean j and variance j2

52. Gaussians for Acoustic Modeling
A Gaussian is parameterized by a mean and a variance:
Different means
P(o|q):
P(o|q) is highest here at mean
P(o|q is low here, very far from mean)
P(o|q) o

53. Multivariate Gaussians
Instead of a single mean  and variance :
Vector of means  and covariance matrix 
Usually assume diagonal covariance
This isn’t very true for FFT features, but is fine for MFCC features

54. Gaussian Intuitions: Size of 
 = [0 0]  = [0 0]  = [0 0]
 = I  = 0.6I  = 2I
As  becomes larger, Gaussian becomes more spread out; as  becomes smaller, Gaussian more compressed
Text and figures from Andrew Ng’s lecture notes for CS229

55. Gaussians: Off-Diagonal
As we increase the off-diagonal entries, more correlation between value of x and value of y
Text and figures from Andrew Ng’s lecture notes for CS229

56. In two dimensions
From Chen, Picheny et al lecture slides

57. In two dimensions
From Chen, Picheny et al lecture slides

58. But we’re not there yet
Single Gaussian may do a bad job of modeling distribution in any dimension:
Solution: Mixtures of Gaussians
Figure from Chen, Picheney et al slides

59. Mixtures of Gaussians M mixtures of Gaussians:
For diagonal covariance:

60. GMMs Summary: each state has a likelihood function parameterized by:
M Mixture weights
M Mean Vectors of dimensionality D
Either
M Covariance Matrices of DxD
Or more likely
M Diagonal Covariance Matrices of DxD
which is equivalent to
M Variance Vectors of dimensionality D

61. Training Mixture Models
Forced Alignment
Computing the “Viterbi path” over the training data is called “forced alignment”
We know which word string to assign to each observation sequence.
We just don’t know the state sequence.
So we constrain the path to go through the correct words
And otherwise do normal Viterbi
Result: state sequence!

62. Modeling phonetic context
W iy r iy m iy n iy

63. “Need” with triphone models

64. Implications of Cross-Word Triphones
Possible triphones: 50x50x50=125,000
How many triphone types actually occur?
20K word WSJ Task (from Bryan Pellom)
Word-internal models: need 14,300 triphones
Cross-word models: need 54,400 triphones
But in training data only 22,800 triphones occur!
Need to generalize models.

65. State Tying / Clustering
[Young, Odell, Woodland 1994]
How do we decide which triphones to cluster together?
Use phonetic features (or ‘broad phonetic classes’)
Stop
Nasal
Fricative
Sibilant
Vowel
lateral

66. State Tying Creating CD phones: Start with monophone, do EM training
Clone Gaussians into triphones
Build decision tree and cluster Gaussians
Clone and train mixtures (GMMs

67. Simple Periodic Waves Characterized by: period: T amplitude A phase 
Fundamental frequency
F0=1/T
1 cycle