1. Laboratory for Experimental ORL KULeuven Perception of major acoustic cues Astrid van Wieringen 5th European Master school on Language and Speech Bonn, 12-16 July 2004

2. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 20042 Content of Tutorial In order to understand why certain speech sounds are not perceived/recognized by hearing-impaired or automatic speech recogniser, one should understand: –major categories + acoustic properties of speech sounds –different types of tests & speech materials –how to assess transmission of robust spectral and temporal cues by means of analytical (phoneme) tests –data collection and analyses hearing loss (with focus on cochlear implants) Practical part: –Test perception of filtered speech sounds

3. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 20043 Speech sounds - major categories Vowel and consonant phonemes are classified in terms of Manner of articulation –concerns how the vocal tract restricts airflow completely stopping of airflow by an occlusion creates a plosive (stop consonant) vocal tract constrictions of varying degree occur in liquids, fricatives, glides and vowels lowering the velum causes nasal sounds Place of articulation refers to the location in the vocal tract Voicing presence/absence of vocal fold vibration

4. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 20044 Manner of articulation of most consonants Stop consonants (plosives): complete closure and subsequent release of a vocal tract obstruction. Pressure build-up followed by burst. Liquids: like vowels, but tongue is used for some degree of obstruction. For /l/ air escapes around the tip of tongue or dorsum. The /r/ has more variable articulation Nasals: a lowering of the velum. Airflow out of the nostrils. In English only nasalized consonants (oral tract completely closed), in French also nasalized vowels (air escapes through oral tract and nasal cavities). Vowels may be nasalized in English, but the distinction is not phonemic (= vowel identity does not change). In French there are pairs of vowels that differ only in the presence or absence of vowel nasalization. Fricatives: narrow constriction in the oral tract (for some language in the pharynx and in the glottis). If the pressure behind the constriction is high enough and the passage sufficiently narrow, airflow becomes fast enough to generate turbulence at the end of the constriction Strident fricatives : noise amplitude is enhanced by airflow striking a surface:  (shy) Affricate= stop + fricative: d  (gin)

5. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 20045 Place of articulation (varies per language)

6. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 20046 Place of articulation Labials –bilabial: if both lips constrict –labiodental: if the lower lip contacts the upper teeth Dental: the tongue tip or blade touches the edge or back of upper teeth –interdental: if the tip protrudes between the upper and lower teeth (‘the’) Alveolar: the tongue tip or blade touches the alveolar ridge Palatals: the tongue blade or dorsum constricts with the hard palate –retroflex: if the tongue tip curls up Velar: the dorsum approaches the soft palate Uvular: the dorsum approaches the uvula Pharyngeal: constriction in the pharynx Glottal: vocal folds close or constrict

7. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 20047 Dutch vowel triangle

8. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 20048 Major acoustic cues of stop consonants /p, t, k, b, d, g/ Phonetic features –Manner: stop (plosive) –Place (bilabial, alveolar, velar) –Acoustic cues Silence (corresponds to the period of oral constriction = stop gap) »Voiced stops: low energy, also called voice bar Burst: corresponds to the articulatory release of the oral constriciton and to aerodynamic release (due to build-up of pressure). Bursts occur in initial and medial position, rarely found in final position. Place of articulation may be signaled by spectrum of burst, but –Transition is also very important. Transition corresponds to the articulatory movement from oral constriction for the stop to the more open tract for a following sound (usually vowel). Easy to identify for voiced than for voiceless sounds. Most important features: stop gap release burst presence/absence of voice onset time transition voicing features

9. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 20049 Duration of stop consonants Stop gap: 50-100 ms Burst: 5-40 ms (a ‘transient’ = disappears immediately, shortest event in speech!) CV (consonant - vowel) and VC (vowel consonant) transitions: 10 - 40 ms. Reflects changes in the vocal tract. Very difficult to measure/analyze such a short event. However, perceptually very important!

10. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200410 Time-signals of Dutch plosives

11. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200411 Time (s) 00.0687982 -0.1792 0.1817 0 Onset of / b / in / aba / Time (s) 00.0817687 -0.2195 0.296 0 Onset of / d / in / ada / Time (s) 00.111791 -0.2101 0.276 0 Onset of / t / in / ata / Time (s) 00.0812925 -0.2187 0.221 0 Onset of / p / in / apa / Initial part of Dutch plosives

12. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200412 Time (s) 00.677007 0 10 4 / aba / Time (s) 00.700023 0 10 4 / apa / Time (s) 00.704014 0 10 4 / ada / Time (s) 00.675011 0 10 4 / ata / Time (s) 00.690023 0 10 4 / aka / Spectrogram of Dutch plosives

13. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200413 Fricatives Phonemes: –voiced / , , ,  / –voiceless: / , , , , ,  / Phonetic features: –manner: frication –place: labiodental, linguadental, alveolar, palatal, glottal Acoustic cues: –voicing –frication noise: noise generated as air is forced through a narrow constriction. Then filtered by the vocal tract. – transitions to and from the vowels due to changes in the vocal tract –sibilants/ stridents have intense noise energy –non sibilants: weak noise energy

14. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200414 Spectrograms of a few Dutch fricatives Time (s) 00.716032 0 10 4 / afa / Time (s) 00.754014 0 10 4 / ava / Time (s) 00.738005 0 10 4 / asa / Time (s) 00.728027 0 10 4 / aza /

15. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200415 Nasals Phonemes: /m, n,  / Phonetic features: –manner: nasal –place: bilabial, alveolar, velar Acoustic features: –murmur: as a result of nasal radiation of acoustical energy. The spectrum is dominated by low-freq. energy (< 500 Hz). Murmur cues of three different nasals are not exactly alike, but difficult as a distinctive cue –transitions: preceding and following vowels will be nasalized. Cues to place of articulation –voicing is always present (except during whispering) Spectrum of nasals reflects a combination of formants and antiformants

16. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200416 Spectrograms of a few Dutch nasals

17. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200417 Glides also called ‘ approximants ’ and semivowels: –gradual articulatory movement –vocal tract narrowed, not closed Phonemes: /j/ & /w/ Phonetic features –Manner: glide or semivowel –Place: palata l or labiovelar Acoustic cues –A relatively slow transition (75-150 ms) –F1 of both sounds starts at very low value (a little higher than for stops) –F2 of /w/: 800 Hz Compare with /b/!!, F3 of /w/: 2200 Hz –F2 of /j/: 2200 Hz (compare wih /d/!!), F3 is 3000 Hz longer glides: vowel-vowel sequences!: –[bi] - [wi]- [ui] and –[du] - [ju] - [iu]

18. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200418 Spectrograms of 2 Dutch glides

19. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200419 Liquids Phonemes: /l/ & /r/ Phonetic features: –Manner: lateral or rhotic –Place: alveolar for /l/, palatal for /r/ Acoustic cues: rather complex: –both relatively fast formant transitions –similarity with glides: well-defined formant structure (less constriction than stops, fricatives, and affricates) –/l/: energy mainly in the low frequencies. Resonances and antiresonances due to divided vocal tract. Resembles /n/. F1: 360 Hz, F2: 1300 Hz, F3: 2700 Hz –/r/: similar for F1 F2 somewhat lower than for /l/ F3 especially lower (1650 Hz). Durations of formant transitions somewhat longer for /r/ than for /l/ –temporal cues: /r/: F1 has a short steady-state + relatively long transition /l/: F1 has a long steady-state + relatively short transition

20. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200420 Spectrograms of 2 Dutch liquids no clear distinction between vowel and consonant F3 of /r/ lower

21. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200421 Speech perception assessment for the hearing-impaired

22. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200422 Speech perception assessment Required for diagnostic purposes monitoring progress in a rehabilitation programme comparison of different speech processing strategies (hearing aids and ochlear implants) understand “limited” technology/number of channels available for hearing impaired or implantees –hearing aid: speech divided into frequency bands. Acoustically enhanced –cochlear implant: acoustical sound is picked up by microphone, analyzed into frequency bands, coded and sent to limited number of electrode pairs in the inner ear (electrical stimulation)

23. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200423 How a cochlear implant works... (MedEL) (1)Sounds are picked up by a microphone and turned into an electrical signal. (2) This signal goes to the speech processor where it is "coded" (turned into a special pattern of electrical pulses). (3) These pulses are sent to the coil and are then transmitted across the intact skin (by radio waves) to the implant. (4) The implant sends a pattern of electrical pulses to the electrodes in the cochlea.(5) The auditory nerve picks up these tiny electrical pulses and sends them to the brain. (6) The brain recognizes these signals as sound.

24. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200424 Tutorial article on cochlear implants that appeared in the IEEE Signal Processing Magazine, pages 101-130, September 1998. Introduction to cochlear implants Philipos C. Loizou

25. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200425 Figure of electrode array in the cochlea... Necessary to ‘map’ (fit) acoustical information to electrical information....

26. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200426 Top: Output of the CIS algorithm for the word ‘som’. Pulse channels reflect the envelopes of the bandpass filter output 100200300400500600700800900 1 2 3 4 5 6 7 8 time (ms) amplitude per channel A

27. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200427 Transmission of AMA & ASA by a CI device

28. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200428 Many types of speech tests to evaluate CI performance –detection of environmental sounds –identification of male/female voice –identification of vowels and consonants (V & C) in nonsense cont. –words –sentences Each type of test triggers a different level of performance. Why is a carefully balanced V & C test important? – /paat/, /pit/, /poot/, etc., or /apa/, /ara/, /ana/, –it gives important information on the transmission of speech features via the implant and hearing aid (e.g. voiced- voicelessness, nasality of /m/ or high frequency frication/turbulence of /s/) analytical: no contextual information therefore, information can guide the fitting of an implant Analytical tests: purpose and performance

29. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200429 speech stimuli should be –carefully pronounced and, if possible, adjusted to the same RMS level (so that other cues are kept in hand) –presented via hard disc of PC, CD or tape (recorded at highest level of quality) –administered to the subject in a quiet room (if presented acoustically) –presented a sufficient number of times to obtain a reliable score –Note: an analytical test does not replace other tests, but it measures speech perception based on auditory information alone. Can be used for several languages. At the Lab. Exp. ORL recordings were made of Dutch vowels and consonants in different contexts. These were carefully selected from different tokens, segmented (with an additional hamming window to avoid on- and offset clicks), equalized in RMS (root mean square) and partly analyzed (with regard to their main spectral and temporal properties). –/aCa/: /p, t, k, b, d, r, l, m, n, s, f, z, v, w, j/. –/pVt/:/oe, ie, i, oo, o, ee, e, u, aa, a/ All speech sounds were analyzed (frequency, duration, energy, …) Choice of test depends on objectives, BUT

30. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200430 Confusion matrix Distribution of errors even more interesting –not random –can be quantified by means of an information transmission algorithm (Miller and Nicely, 1955)

31. Lab Exp ORL KULeuven Astrid van Wieringen 12-16 July 200431 Effect of filtering Loss of auditory information can be examined in normal-hearing persons by filtering away acoustical information: to allow certain frequencies to be transmitted while attenuating others. –a high-pass filter allow all components above a cutoff frequency to be transmitted –a band-pass filter allows frequencies within a certain band to pass –low-pass filter allow all components below a cutoff frequency to be transmitted Demonstration of loss of acoustical cues!