1. CSE 551/651: Structure of Spoken Language Lecture 1: Visualization of the Speech Signal, Introductory Phonetics John-Paul Hosom Fall 2005

2. Structure of Spoken Language : Hosom 2 Visualization of the Speech Signal Most common representations: Time-domain waveform Energy Pitch contour Spectrogram (power spectrum)

3. Structure of Spoken Language : Hosom 3 Visualization of the Speech Signal: Time-Domain Waveform Time-domain waveform is a signal recorded directly from microphone, with time on horizontal axis and amplitude on vertical axis. “Variations in air pressure in the form of sound waves move through the air somewhat like ripples on a pond. … A graph of a sound wave is very similar to a graph of the movements of the eardrum.” [Ladefoged, p. 184] “Sound originates from the motion or vibration of an object. This motion is impressed upon the surrounding medium (usually air) as a pattern of changes in pressure. … The sound generally weakens as it moves away from the source and also may be subject to reflections and refractions…” [Moore, p. 2]

4. Structure of Spoken Language : Hosom 4 Visualization of the Speech Signal: Time-Domain Waveform Vertical axis: amplitude, relative sound pressure typical unit:  Pa (micro-pascals) (digital signal usually unitless) quantization (-32768 to 32767) Horizontal axis: time typical unit: msec (milliseconds) sampling (8000, 16000, 44.1K samp/sec)

5. Structure of Spoken Language : Hosom 5 Visualization of the Speech Signal: Energy “Energy” or “Intensity”: intensity is sound energy transmitted per second (power) through a unit area in a sound field. [Moore p. 9] intensity is proportional to the square of the pressure variation [Moore p. 9] normalized energy = = intensity x n = signal x at time sample n N = number of time samples

6. Structure of Spoken Language : Hosom 6 Visualization of the Speech Signal: Energy “Energy” or “Intensity”: human auditory system better suited to relative scales: energy (bels) = energy (decibels, dB) = I 0 is a reference intensity… if the signal becomes twice as powerful (I 1 /I 0 = 2), then the energy level is 3 dB (3.01023 dB to be more precise) Typical value for I 0 is 20  Pa. 20  Pa is close to the average human absolute threshold for a 1000-Hz sinusoid.

7. Structure of Spoken Language : Hosom 7 Visualization of the Speech Signal: Energy What is a good value of N? Depends on information of interest: N=1 msec N=5 msec N=20 msec N=80 msec

8. Structure of Spoken Language : Hosom 8 Visualization of the Speech Signal: Power Spectrum What makes one phoneme, /aa/, sound different from another phoneme, /iy/? Different shapes of the vocal tract… /aa/ is produced with the tongue low and in the back of the mouth; /iy/ is produced with the tongue high and toward the front. The different shapes of the vocal tract produce different “resonant frequencies”, or frequencies at which energy in the signal is concentrated. (Simple example of resonant energy: a tuning fork may have resonant frequency equal to 440 Hz or “A”). Resonant frequencies in speech (or other sounds) can be displayed by computing a “power spectrum” or “spectrogram,” showing the energy in the signal at different frequencies.

9. Structure of Spoken Language : Hosom 9 Visualization of the Speech Signal: Power Spectrum A time-domain signal can be expressed in terms of sinusoids at a range of frequencies using the Fourier transform: where x(t) is the time-domain signal at time t, f is a frequency value from 0 to 1, and X(f) is the spectral-domain representation. note:

10. Structure of Spoken Language : Hosom 10 Visualization of the Speech Signal: Power Spectrum Since samples are obtained at discrete time steps, and since only a finite section of the signal is of interest, the discrete Fourier transform is more useful: in which x(k) is the amplitude at time sample k, n is a frequency value from 0 to N-1, N is the number of samples or frequency points of interest, and X(n) is the spectral-domain representation of x(k). Note that we assume that that the series outside the range (0, N-1) is “extended N-periodic,” that is, x k = x k+N for all k.

11. Structure of Spoken Language : Hosom 11 Visualization of the Speech Signal: Power Spectrum The sampling frequency is the rate at which samples are recorded; e.g. 8000 Hz = 8000 samples per second. Shannon’s Sampling Theorem states that a continuous signal must be discretely sampled with at least twice the frequency of the highest frequency present in the signal. So, the signal must not contain any data above F samp /2 (the Nyquist frequency). If it does, use a low-pass filter to remove these higher frequencies. Because the signal is assumed to be periodic over length N, but this assumption is usually false, then the signal is weighted with a window so that both edges of the signal taper toward zero: Hamming window:

12. Structure of Spoken Language : Hosom 12 Visualization of the Speech Signal: Power Spectrum The magnitude and phase of the spectral representation are: Phase information is generally considered not important in understanding speech, and the energy (or power) of the magnitude of F(n) on the decibel scale provides most relevant information: Note: usually don’t worry about reference intensity I 0 (assume a value of 1.0); the signal strength (in  Pa) is unknown anyway. absolute value of complex number

13. Structure of Spoken Language : Hosom 13 Visualization of the Speech Signal: Power Spectrum The power spectrum can be plotted like this (vowel /aa/): time- domain amplitude spectral power (dB) (512 samp) 0 Hz4000 Hz 73 dB frequency (Hz)

14. Structure of Spoken Language : Hosom 14 Visualization of the Speech Signal: Power Spectrum If the signal is periodic and the number of samples in the window is large enough, harmonics are seen: periodic signal/aa/periodic signal /aa/ aperiodic signal /sh/ 128 samples2048 samples 2048 samples (frequency range is 0 to 4000 Hz in all plots) A harmonic is a strong energy component at an integer multiple of the fundamental frequency (pitch), F0.

15. Structure of Spoken Language : Hosom 15 Visualization of the Speech Signal: Formants Note that the resonant frequencies, or formants, for the two vowels /aa/ and /iy/ can be identified in the spectra. For recognition of phonemes, the spectral envelope is important (envelope = shape of spectrum without harmonics) /aa/ 2048 samples /iy/ 2048 samples ? envelope ? 0 1K 2K 3K 4K

16. Structure of Spoken Language : Hosom 16 Visualization of the Speech Signal: Formants The harmonics, which are dependent on F 0, are not significantly related to the resonant frequencies, which are dependent on the vocal tract shape (or phoneme) 0 1K 2K 3K 4KHz /aa/ F 0 =80Hz /aa/ F 0 =164Hz

17. Structure of Spoken Language : Hosom 17 Visualization of the Speech Signal: Formants These formants can be modeled by a “damped sinusoid”, which has the following representations: where S(f) is the spectrum at frequency value f, A is overall amplitude, f c is the center frequency of the damped sine wave, and  is a damping factor. [Olive, p. 48, 58] time (msec) power (dB) amplitude frequency (Hz) center freq. f c  0 dB 0

18. Structure of Spoken Language : Hosom 18 Visualization of the Speech Signal: Formants The bandwidth is defined as the width of the spectral peak measured at the point where the linear spectral magnitude value is ½ the maximum value. A reduction of the signal by a factor of 2 is equivalent to a 3 dB change. power (dB) frequency (Hz) bandwidth 0 dB 3 dB Also, the resonator must have a value of 0 dB at 0 Hz.

19. Structure of Spoken Language : Hosom 19 Visualization of the Speech Signal: Formants Formants are specified by a frequency, F, and bandwidth, B. A neutral vowel (/ax/) theoretically has formants at 500 Hz, 1500 Hz, 2500 Hz, 3500 Hz, etc. The first formant is called F 1, the second is called F 2, etc. (The fundamental frequency, or pitch, is F 0.) F 1, F 2, and sometimes F 3 are usually sufficient for identifying vowels. Formants can be thought of as filters, which act on the source waveform. For vowels, the source waveform is air pushed through the vibrating vocal folds. Energy is lost (hence a damped sinusoid model) by sound absorption in the mouth. A digital model of a formant can be implemented using an infinite-impulse response (IIR) filter.

20. Structure of Spoken Language : Hosom 20 Visualization of the Speech Signal: Excitation/Source The vocal-fold vibration source looks like this: (Note: there are some gross simplifications here… we’ll go into more detail later in the course.) In fricatives and other unvoiced speech, the source is turbulent air: time (msec) amplitude frequency (Hz) -6 dB/octave power (dB) frequency (Hz) flat slope power (dB) time (msec) amplitude

21. Structure of Spoken Language : Hosom 21 Visualization of the Speech Signal: Pre-Emphasis Because the source for voiced sounds decreases at –6 dB/octave, a simple filter can be used to increase the spectral tilt by +6 dB/octave, thereby making voiced sounds spectrally flat and easier to visualize. (NOTE: unvoiced sounds then have spectral slope of + 6 dB/octave) frequency (Hz) 0 dB/octave frequency (Hz) power (dB) -6 dB/octave where x(n) is the time-domain speech signal at sample number n, and x(n) is the pre-emphasized speech signal at sample n.

22. Structure of Spoken Language : Hosom 22 Visualization of the Speech Signal: Spectrograms Many power spectra can be plotted over time, creating a “spectrogram” or “spectrograph” (pre-emphasis = 0.97): /aa/ freq (Hz) amp /iy/ freq (Hz) amp time (msec) (FFT size = 10 msec)

23. Structure of Spoken Language : Hosom 23 Visualization of the Speech Signal: Spectrograms The FFT window size has a large impact on visual properties: /aa/ freq (Hz) amp /aa/ freq (Hz) “wideband” = small time window = small FFT size “narrowband” = large time window = large FFT size (FFT size = 5 msec) (FFT size = 33 msec)

24. Structure of Spoken Language : Hosom 24 Spectrogram Reading: Vowels 12 English vowels (not all are phonemic), 8 or 9 phonemic vowels: /iy/beet(front, high, unrounded, tense) /ih/bit(front, high, unrounded, lax) /eh/bet(front, mid, unrounded, lax) /ae/bat(front, low, unrounded, lax) /ix/ * roses(back, high, unrounded, lax) (subst. /ih/) /ux/ * suit(back, high, rounded, lax) (subst. /uw/) /ax/ * above(back/central, mid, unrounded, lax) (subst. /ah/) /uw/boot(back, high, rounded, tense) /uh/book(back, high, rounded, lax) /ah/above(back/central, mid, unrounded, lax) /ao/caught(back, low, rounded, tense)(subst. /aa/) /aa/father(back, low, unrounded, tense) * these vowels are more centralized and shorter in duration

25. Structure of Spoken Language : Hosom 25 Spectrogram Reading: Vowels 6 English diphthongs: /ey/bay(front, mid  high, unrounded, tense) /ay/bye(back  front, low  high, unrounded, tense) /oy/boy(back  front, mid  high, rounded  unrounded, tense) /yu/beauty(front  back, high, unrounded  rounded, tense) /aw/about(back, mid  high, unrounded  rounded, tense) /ow/boat(back, mid, unrounded  rounded, tense)

26. Structure of Spoken Language : Hosom 26 Spectrogram Reading: Vowels Vowel formant frequencies (averages for English, males only): *from Ladefoged, p. 193

27. Structure of Spoken Language : Hosom 27 Spectrogram Reading: Vowels Vowel formant frequencies:

28. Structure of Spoken Language : Hosom 28 Spectrogram Reading: Vowels Vowel formants (averages for English, male vs. female): *from Peterson, G.E., and Barney, H.L. (1952). "Control methods used in the study of vowels", Journal of the Acoustical Society of America, 24,175-184.

29. Structure of Spoken Language : Hosom 29 Spectrogram Reading: Vowels Vowel formants, Peterson and Barney data:

30. Structure of Spoken Language : Hosom 30 Spectrogram Reading: Vowels Ratios of 1 st and 2 nd formant, from Miller (1989) based on Peterson and Barney (1952) data:

31. Structure of Spoken Language : Hosom 31 Spectrogram Reading: Diphthongs Diphthongs have characteristic formant movements: /ay/ /aw/ /ow/ /oy/ /yu/ /ei/

32. Structure of Spoken Language : Hosom 32 Spectrogram Reading: Vowels

33. Structure of Spoken Language : Hosom 33 Spectrogram Reading: Vowels

34. Structure of Spoken Language : Hosom 34 Spectrogram Reading: Vowels

35. Structure of Spoken Language : Hosom 35 Spectrogram Reading: Vowels

36. Structure of Spoken Language : Hosom 36 Spectrogram Reading: Vowels

37. Structure of Spoken Language : Hosom 37 Spectrogram Reading: Vowels

38. Structure of Spoken Language : Hosom 38 Spectrogram Reading: Vowels

39. Structure of Spoken Language : Hosom 39 Spectrogram Reading: Vowels

40. Structure of Spoken Language : Hosom 40 Spectrogram Reading: Diphthongs

41. Structure of Spoken Language : Hosom 41 Spectrogram Reading: Diphthongs

42. Structure of Spoken Language : Hosom 42 Spectrogram Reading: Diphthongs

43. Structure of Spoken Language : Hosom 43 Spectrogram Reading: Diphthongs

44. Structure of Spoken Language : Hosom 44 Spectrogram Reading: Diphthongs

45. Structure of Spoken Language : Hosom 45 Spectrogram Reading: Diphthongs