2. Spectral Analysis March 3, 2016

3. Mini-Rant I have succeeded in grading your course project reports. Things to keep in mind: A table of stop phonemes is a theory about how your language works. Your job is to back up that theory with evidence. Minimal pairs are not that important, but examples are. You are a better linguist than your consultant is. You cannot expect your consultant to know what a “minimal pair” is, or to be able to produce them on demand for you. Your consultant may very well say things about their language that are completely uninformed and wrong.

4. Mini-Rant, part II Do not tell me about the writing system for your language. Your job is not to find out how the language is written, but how it is spoken. Be conservative with your conclusions. Only tell me what can be justified with evidence from the data you’ve collected. The process is more important than the product. I would rather that you get comfortable with not knowing what the “right” answer is than for you to tell me what somebody else thinks the right answer is.

8. Moving On… Don’t forget: the next course project report is due on Tuesday! On Tuesday, we will also start adding mystery spectrograms into the mix. …and I will return your mid-terms to you. Today’s goal: let’s learn where spectrograms come from.

9. The Source The complex wave emitted from the glottis during voicing= The source of all voiced speech sounds. In speech (particularly in vowels), humans can shape this spectrum to make distinctive sounds. Some harmonics may be emphasized... Others may be diminished (damped) Different spectral shapes may be formed by particular articulatory configurations....but the process of spectral shaping requires the raw stuff of the source to work with.

10. Spectral Shaping Examples Certain spectral shapes seem to have particular vowel qualities.

11. Spectrograms A spectrogram represents: Time on the x-axis Frequency on the y-axis Intensity on the z-axis

12. Real Vowels

13. Ch-ch-ch-ch-changes Check out some spectrograms of sinewaves which change frequency over time:

14. The Whole Thing What happens when we put all three together? This is an example of sinewave speech.

15. The Real Thing Spectral change over time is the defining characteristic of speech sounds.  It is crucial to understand spectrographic representations for the acoustic analysis of speech.

16. Life’s Persistent Questions How do we get from here: To here? Answer: Fourier Analysis

17. Fourier’s Theorem Joseph Fourier (1768-1830) French mathematician Studied heat and periodic motion His idea: any complex periodic wave can be constructed out of a combination of different sinewaves. The sinusoidal (sinewave) components of a complex periodic wave = harmonics

18. Fourier Analysis Building up a complex wave from sinewave components is straightforward… Breaking down a complex wave into its spectral shape is a little more complicated. In our particular case, we will look at: Discrete Fourier Transform (DFT) Also: Fast Fourier Transform (FFT) is used often in speech analysis Basically a more efficient, less accurate method of DFT for computers.

19. Spectral Slices The first step in Fourier Analysis is to window the signal. I.e., break it all up into a series of smaller, analyzable chunks. This is important because the spectral qualities of the signal change over time. a “window” Check out the typical window length in Praat.

20. The Basic Idea For the complex wave extracted from each window... Fourier Analysis determines the frequency and intensity of the sinewave components of that wave. Do this about 1000 times a second, turn the spectra on their sides, and you get a spectrogram.

21. Possible Problems What would happen if a waveform chunk was windowed like this? Remember, the goal is to determine the frequency and intensity of the sinewave components which make up that slice of the complex wave.

22. The Usual Solution The amplitude of the waveform at the edges of the window is normally reduced... by transforming the complex wave with a smoothing function before spectral analysis. Each function defines a particular window type. For example: the “Hanning” Window

23. There are lots of different window types... each with its own characteristic shape Hamming BartlettGaussian HanningWelchRectangular

24. Window Type Ramifications Play around with the different window types in Praat.

25. Ideas Once the waveform has been windowed, it can be boiled down into its component frequencies. Basic strategy: Determine whether the complex wave correlates with sine (and cosine!) waves of particular frequencies. Correlation measure: “dot product” = sum of the point-by-point products between waves. Interesting fact: Non-zero correlations only emerge between the complex wave and its harmonics! (This is Fourier’s great insight.)

26. A Not-So-Complex Example Let’s build up a complex wave from 8 samples of a 1 Hz sine wave and a 4 Hz cosine wave. Note: our sample rate is 8 Hz. 1 2 3 4 5 6 7 8 A1 Hz 0.707 1.707 0 -.707 -1 -.707 B4 Hz 1 -1 1 -1 1 -1 1 -1 CSum: 1 -.293 2 -.293 1 -1.707 0 -1.707 Check out a visualization.

27. Correlations, part 1 Let’s check the correlation between that wave and the 1 Hz sinewave component. 1 2 3 4 5 6 7 8 CSum: 1 -.293 2 -.293 1 -1.707 0 -1.707 A1 Hz: 0.707 1.707 0 -.707 -1 -.707 C*ADot: 0 -.207 2 -.207 0 1.207 0 1.207 The sum of the products of each sample is 4. This also happens to be the dot product of the 1 Hz wave with itself. = its “power”

28. Correlations, part 2 Let’s check the correlation between the complex wave and a 2 Hz sinewave (a non-component). 1 2 3 4 5 6 7 8 CSum: 1 -.293 2 -.293 1 -1.707 0 -1.707 D2 Hz: 0 1 0 -1 0 1 0 -1 C*DDot: 0 -.293 0.293 0 -1.707 0 1.707 The sum of the products of each sample is 0.  We now know that 2 Hz was not a component frequency of the complex wave.

29. Correlations, part 3 Last but not least, let’s check the correlation between the complex wave and the 4 Hz cosine wave. 1 2 3 4 5 6 7 8 CSum: 1 -.293 2 -.293 1 -1.707 0 -1.707 B4 Hz 1 -1 1 -1 1 -1 1 -1 C*BDot: 1.293 2.293 1 1.707 0 1.707 The sum of the products of each sample is 8. Yes, 8 happens to be the dot product of the 4 Hz wave with itself. its “power”

30. Mopping Up Our component analysis gave us the following dot products: C*A = 4(A = 1 Hz sinewave) C*D = 0(D = 2 Hz sinewave) C*B = 8(B = 4 Hz cosine wave) We have to “normalize” these products by dividing them by the power of the “reference” waves: power (A) = A*A = 4  C*A/A*A = 4/4 = 1 power (D) = D*D = 4  C*D/D*D = 0/4 = 0 power (B) = B*B = 8  C*B/B*B = 8/8 = 1 These ratios are the amplitudes of the component waves.

31. Let’s Try Another Let’s construct another example: 1 Hz sinewave + a 4 Hz cosine wave with half the amplitude. 1 2 3 4 5 6 7 8 A1 Hz 0.707 1.707 0 -.707 -1 -.707.5*B4 Hz.5 -.5.5 -.5.5 -.5.5 -.5 ESum:.5.207 1.5.207.5 -1.207 -.5 -1.207 Let’s check the 1 Hz wave first: ESum:.5.207 1.5.207.5 -1.207 -.5 -1.207 A1 Hz 0.707 1.707 0 -.707 -1 -.707 E*ADot: 0.146 1.5.146 0.854.5.854 Sum = 4

32. Yet More Dots Another example: 1 Hz sinewave + a 4 Hz cosine wave with half the amplitude. Now let’s check the 4 Hz wave: ESum:.5.207 1.5.207.5 -1.207 -.5 -1.207 B4 Hz 1 -1 1 -1 1 -1 1 -1 E*BDot:.5 -.207 1.5 -.207.5 1.207 -.5 1.207 The sum of these products is also 4. = half of the power of the 4 Hz cosine wave.  The 4 Hz component has half the amplitude of the 4 Hz cosine reference wave. (we know the reference wave has amplitude 1)

33. Mopping Up, Part 2 Our component analysis gave us the following dot products: E*A = 4(A = 1 Hz sinewave) E*B = 4(B = 4 Hz cosine wave) Let’s once again normalize these products by dividing them by the power of the “reference” waves: power (A) = A*A = 4  E*A/A*A = 4/4 = 1 power (B) = B*B = 8  E*B/B*B = 4/8 =.5 These ratios are the amplitudes of the component waves. The 1 Hz sinewave component has amplitude 1 The 4 Hz cosine wave component has amplitude.5

34. Footnote Sinewaves and cosine waves are orthogonal to each other.  The dot product of a sinewave and a cosine wave of the same frequency is 0. 1 2 3 4 5 6 7 8 Asin 0.707 1.707 0 -.707 -1 -.707 Fcos 1.707 0 -.707 -1 -.707 0.707 A*FDot: 0.5 0 -.5 0.5 0 -.5 However, adding cosine and sine waves together simply shifts the phase of the complex wave. Check out different combos in Praat.

35. Problem #1 For any given window, we don’t know what the phase shift of each frequency component will be. Solution: 1.Calculate the correlation with the sinewave 2.Calculate the correlation with the cosine wave 3.Combine the resulting amplitudes with the pythagorean theorem: Take a look at the java applet online: http://www.phy.ntnu.edu/tw/ntnujava/index.php?topic=148

36. Sine + Cosine Example Let’s add a 1 Hz cosine wave, of amplitude.5, to our previous combination of 1 Hz sine and 4 Hz cosine waves. 1 2 3 4 5 6 7 8 C1+4: 1 -.293 2 -.293 1 -1.707 0 -1.707.5*Fcos.5.353 0 -.353 -.5 -.353 0.353 GSum: 1.5.06 2 -.646.5 -2.06 0 -1.353 Let’s check the 1 Hz sine wave again: GSum: 1.5.06 2 -.646.5 -2.06 0 -1.353 A1 Hz 0.707 1.707 0 -.707 -1 -.707 G*ADot: 0.043 2 -.457 0 1.457 0.957 Sum = 4

37. Sine + Cosine Example Now check the 1 Hz cosine wave: GSum: 1.5.06 2 -.646.5 -2.06 0 -1.353 F1 Hz 1.707 0 -.707 -1 -.707 0.707 G*FDot: 1.5.043 0.457 -.5 1.457 0 -.957 Sum = 2 Sinewave component amplitude = 4/4 = 1 Cosine wave component amplitude = 2/4 =.5 Total amplitude = Check out the amplitude of the combo in Praat.

38. In Sum To perform a Fourier analysis on each (smoothed) chunk of the waveform: 1.Determine the components of each chunk using the dot product-- Components yield a dot product that is not 0 Non-components yield a dot product that is 0 2.Normalize the amplitude values of the components  Divide the dot products by the power of the reference wave at that frequency 3.If there are both sine and cosine wave components at a particular frequency: Combine their amplitudes using the Pythagorean theorem