Speech acoustics and phonetics Louis C.W. Pols Institute of Phonetic Sciences (IFA) Amsterdam Center for Language and Communication (ACLC) NATO-ASI “Dynamics of Speech Production and Perception” Il Ciocco, Tuscany, Italy, July 1, 2002

July 1st, 2002Speech acoustics and phonetics, Il Ciocco2 Overview Dynamics in speech acoustics Contour modeling (mainly formants) Aspects of spectral undershoot Modeling V and C reduction Phonetic knowledge from speech corpora IFA, CGN, TIMIT, found speech Conclusions

July 1st, 2002Speech acoustics and phonetics, Il Ciocco4 Dynamics in speech acoustics Dynamics is the norm, not stationarity articulatory efficiency Dynamics is everywhere generally no word boundaries in speech deletion of words, syllables, phonemes; insertion within/between word coarticulation/assimilation vowel and consonant reduction Acoustic manifestations segment duration, F0, loudness, spectral quality

July 1st, 2002Speech acoustics and phonetics, Il Ciocco5 Dynamics is the norm The speaker speaks as sloppily as the listeners allow him to do in communication communicative efficiency Articulatory vs. perceptual efficiency do spectral transitions facilitate or hamper perception? —> see other presentation Speaker flexibility; speaking style (clear vs. sloppy); speaking rate

July 1st, 2002Speech acoustics and phonetics, Il Ciocco6 Dynamics is everywhere Deletion ‘bread and butter’ /brEmbY3/ ‘Amsterdam’ (Du) ‘koninklijke’ (Du) Insertion homorganic glide insertion: ‘die een’ (Du) Degemination ‘is zichtbaar’ (Du) /Is zIxtbar/ —>/IsIxbar/ Reduction, coarticulation, assimilation

July 1st, 2002Speech acoustics and phonetics, Il Ciocco7 Acoustic manifestations pitch, loudness, formant, component contours contour stylization (e.g., pitch in praat)praat contour modeling n-th degree curve fitting(D.van Bergem) Legendre polynomials)(R.van Son) 16 points per segment) (phoneme) segmentation by hand (time consuming; non-consistent) automatically (via forced phoneme recognition and a pronunciation lexicon with alternatives; systematic errors)

July 1st, 2002Speech acoustics and phonetics, Il Ciocco8 Contour modeling allows modeling of specific phenomena pitch accentuation (vs. vowel onset) reduction, centralization, undershoot allows generation of stimuli for perc. expts. phoneme identification in extending context 2-alternatives forced choice identif. of continua discrimination, RT allows statistics on large speech corpora TIMIT, CGN, IFA-corpus, Switchboard

July 1st, 2002Speech acoustics and phonetics, Il Ciocco9 Static vs. dynamic V recogn. see Weenink (2001) “Vowel normalizations with the TIMIT acoustic phonetic speech corpus”, IFA Proc. 24, males, both train & test sent. of TIMIT 35,385 vowel segments, hand segmented 13 monophthongeal vowel categories 1-Bark bandfilter anal. (18), intensity. normal. 3 frames per segment: central and 25 ms L/R

July 1st, 2002Speech acoustics and phonetics, Il Ciocco10 Some results Vowel classif. (%) with discriminant functions Condition# ItemsStatic 1 frame Dynamic 3 frames Original35, x13x(1…25) speaker normalized 35, V centers per speaker 5, x speaker normalized 5,

July 1st, 2002Speech acoustics and phonetics, Il Ciocco11 Formant tracks / speaking rate Ph.D. thesis Rob van Son (1993) “Spectro-temporal features of vowel segments” see also Speech Comm. 13, (Pols & vSon) 850-words text, read at normal and fast rate hand segmentation of 7 most freq. V + schwa formant tracks via 16 points per segm. or 5 Legendre polynomials influence of rate, V-dur., context, sent. acc. evidence for duration-controlled undershoot?

July 1st, 2002Speech acoustics and phonetics, Il Ciocco12 Some results no differences for F1/F2 in vowel center for normal- or fast-rate speech; only some over- all rise in F1 for fast rate (irrespective of V) same formant track shape (normalized to 16 points) for normal- or fast-rate speech same results when using the more elaborate Legendre polynomials Concl.: changes in V-duration do not change the amount of undershoot —> active control of articulation speed

July 1st, 2002Speech acoustics and phonetics, Il Ciocco13 Formant representations zeroth order Legendre Legendre polynomial coefficients (mean F i in vowel segment) second order polynomials (axes reversed) e e

July 1st, 2002Speech acoustics and phonetics, Il Ciocco14 Modeling vowel reduction Ph.D. thesis Dick van Bergem (1995) “Acoustic and lexical vowel reduction” see also Speech Communication 16, lexical V reduction Fr /betõ/ vs. Du acoustic V reduction /banan, bAnan, f(sent. acc., w. str., w. class): can-candy-canteen coarticulatory effects on the schwa C 2 V- and VC 2 -type nonsense words perceptual effects (full V or schwa, f.i. ‘ananas’)

July 1st, 2002Speech acoustics and phonetics, Il Ciocco15 Some results The schwa is not just a centralized vowel but something that is completely assimilated with its phonemic context t-nw-l

July 1st, 2002Speech acoustics and phonetics, Il Ciocco16 Modeling consonant reduction Sp. Comm. (1999) 28, (vSon & Pols) 20 min. speech, both spontaneous and read 2 x 791 similar VCV; hand segmented 5 aspects of V and C reduction related to coarticulation: F2 slope differences at CV- vs. VC-boundaries; F2 locus equations (F2 onset vs. F2 target) related to speaking effort: duration; spectral COG (mean freq.); V-C sound energy differences

July 1st, 2002Speech acoustics and phonetics, Il Ciocco17 Some results V markedly reduced in spontaneous speech lower F2-slope diff. in spontaneous speech —> decrease in articulation speed no systematic effect on F2 locus equation; V onsets and targets change in concert —> any V reduction mirrored by comparable change in C spont. sp.: V and C shorter; lower COG —> decrease in vocal and articulatory effort

July 1st, 2002Speech acoustics and phonetics, Il Ciocco18 Access to large corpora more, and more realistic, data phonetic knowledge via statistical analyses f.i. highly accessible IFA-corpus (free, SQL) see “Structure and access of the open source IFA-corpus”, IFA Proc. 24, (vSon & Pols) on-line 4 M/4F speakers, 5.5 hrs of speech from informal to read + sent., words, syllables ~ 50Kwords segm. and labeled at phoneme level

July 1st, 2002Speech acoustics and phonetics, Il Ciocco19 Some results speech + annot. + meta data: relational DB realization of final n, f.i. Du ‘geven’ Informal 5, Retelling 6, LF HF Narr. story 14, Sentences 14, Pseudo-sent 2, All43, ,2711,73036 Read

July 1st, 2002Speech acoustics and phonetics, Il Ciocco20 Spoken Dutch Corpus (CGN) 10 M words, 1,000 hrs of speech variety of styles, incl. telephone speech adult Dutch and Flemish speakers for linguistic and technological research see various LREC and ICSLP papers (2002) see also fully transcribed: orthogr., POS, lemmas partly transcr.: phonemic, prosodic, syntactic

July 1st, 2002Speech acoustics and phonetics, Il Ciocco21 TIMIT popular DB in acoustic phonetics and ASR also telephone version (NTIMIT) hand segmented & labeled at phoneme level 438 males, 192 females (8 dialect regions) 10 sent./sp. (2 fixed, 1 phon. compact, 7 diverse) sa1: “She had her dark suit in greasy wash water all year” includes separate test data (112 M, 56 F) e.g. Ph.D thesis X. Wang (1997) “Incorporating knowledge on segmental duration in HMM-based continuous speech recognition”

July 1st, 2002Speech acoustics and phonetics, Il Ciocco22 Useful info: durational variability Adopted from Wang (1998) normal rate=95 primary stress=104 word final=136 utterance final=186 overall average=95 ms

normalized phone durationspeaking rate all 3,696 training sent. (sx + si) of TIMIT training set 0

July 1st, 2002Speech acoustics and phonetics, Il Ciocco24 ‘found’ speech DARPA-LVSR community rather ambitious Broadcast News (BN), Sp.Comm. 37 (2002) < ’95 WSJ NAB read sp Market place 1996 F0-F5, FX partitioned hrs test unpartit non Engl. speech also < 10x RT audio training data 100 hrs10 hrs55 hrs+ 50 hrs+ 100 hrs text (for LM)430 K122 M540 M> 900 M best % WER on test set 27.0 %27.1 % 1:46 hrs 16.2 % 3 hrs 13.5 —>16.1 % 3 hrs (10xRT) For Proc. DARPA Workshops, see

July 1st, 2002Speech acoustics and phonetics, Il Ciocco25 Articul.-acoustic features in ASR “A Dutch treatment of an elitist approach to articulatory-acoustic feature classification”, Proc. Eurospeech-2001, (M. Wester et al.) “Integrating articulatory features into acoustic models for speech recognition”, Phonus 5, (K. Kirchhoff, 2000) “An overlapping-feature-based phonological model incorporating linguistic constraints: Applications to speech recognition”, JASA 111 (2), (J. Sun & L. Deng, 2002)

July 1st, 2002Speech acoustics and phonetics, Il Ciocco26 Conclusions examples of dynamics in speech acoustics going from formal to informal speech: less dynamics, more reduction (artic. guided) undershoot vs. speaking style sloppiness or articulatory limits? functionality of dynamics? —> other paper systematicity of dynamics? easing ASR, rules for TTS, acquiring knowledge?