1. Acoustic and Linguistic Characterization of Spontaneous Speech Masanobu Nakamura, Koji Iwano, and Sadaoki Furui Department of Computer Science Tokyo Institute of Technology Tokyo, Japan

2. 2 Introduction(1/2) Background  Present speech recognition technology  High recognition accuracy for read speech  Rather poor accuracy for spontaneous speech  Improvement of recognition accuracy for spontaneous speech is necessary.  What are the differences between spontaneous and read speech?  Why is the recognition accuracy for spontaneous speech low? What are differences?

3. 3 Introduction(2/2) Goals  Statistical and quantitative analysis of acoustic and linguistic differences between spontaneous and read speech.  Investigation of acoustic and linguistic characteristics which affect speech recognition performance in spontaneous speech.

4. 4 Corpus of Spontaneous Japanese (CSJ) A large-scale spontaneous speech corpus  Roughly 7M words (morphemes) with a total speech length of 650 hours  Orthographic and phonetic transcription are manually given. Speaking styles Academic presentations(AP)  Live recordings of academic presentations  The fields of Engineering, social science, and humanities Extemporaneous presentations(EP)  Studio recordings of paid layman speakers’ speech  Small audience and relatively relaxed atmosphere  More informal than AP Dialogue speech(D)  Interview, task oriented dialogue, and free dialogue Read speech(R)  Reading transcription of AP or EP by the same speaker

5. 5 Disfluency ratio Filled pauses (F), word fragments (W), and reduced articulation or mispronunciation (M) Approximately one-tenth of the words are disfluencies in the spontaneous speech in the CSJ. The ratio of F is significantly higher than that of W and M.

6. Acoustic characteristics

7. 7 Acoustic feature extraction 1. 39-dimensional feature vectors  12-dimensional MFCC, log-energy, and their first and second derivatives  25 ms-length window shifted every 10 ms  CMS is applied to each utterance. 2. HMMs  Mono-phone HMMs with a single Gaussian mixture  Left-to-right topology with three self-loops  Trained using samples of every combination of phonemes, speakers, and utterance styles 3. Acoustic features for each phoneme  Mean and variance vectors of 12-dimensional MFCC at the 2nd state of the HMM Target phonemes  31 Japanese phonemes (10 vowels and 21 consonants) Vowel /a, i, u, e, o, a:, i:, u:, e:, o:/ Conso- nant /w, y, r, p, t, k, b, d, g, j, ts, ch, z, s, sh, h, f, N, N:, m, n/

8. 8 Reduction ratio Quantitative analysis of the spectral space reduction for spontaneous speech Definition   p ( X ) is the mean vector of a phoneme p uttered with a speaking style X.   p ( R ) is the mean vector of read speech.  Av: average over all phonemes  || || : Euclidean norm / distance Speaking style X Read speech Center of the distribution of all phonemes Phoneme p

9. 9 Reduction ratio averaged over 10 speakers MFCC space is reduced for almost all the phonemes, and this is most significant for dialogue utterances. red p (X) = 1

10. 10 Reduction ratio averaged over vowels and consonants Reduction of the distribution of spontaneous speech in comparison with read speech is observed for all the speaking styles, and this is most significant for dialogue speech.

11. 11 Between-phoneme distances The reduction of the MFCC distance between each phoneme pair is measured by using Mahalanobis distance. Phoneme cepstrum Mahalanobis distance between each phoneme pair a r n u m k ar n u m k MFCC space

12. 12 Mahalanobis distance Mahalanobis distance D ij ( X ) between phoneme i and j :  K : dimension of MFCC vector ( K = 12)   ik and  ik 2 : k th elements of the mean and variance vector of MFCC for phoneme i uttered with a speaking style X.

13. 13 Cumulative frequency of distribution of Mahalanobis distances Mahalanobis distances between every phoneme pair for each speaking style Mahalanobis distance between phonemes decreases as the spontaneity of utterances increases. The more spontaneous the utterances become, the more reduced the cepstrum space becomes. Increase of spontaneity

14. 14 Relationship between phoneme distances and phoneme accuracy (1/2) Investigation of relationship between mean phoneme distances and phoneme recognition accuracy Acoustic model  A common model for all speaking styles  Trained on the data from 100 males and 100 females for AP and 150 males and 150 females for EP (about 2M phoneme samples, respectively) Language model  Phoneme network constrained by phoneme-class probabilities

15. 15 Strong correlation between mean phoneme distance and phoneme accuracy Reduction of the distances between phonemes is a major factor contributing to the degradation of spontaneous speech recognition accuracy. Relationship between phoneme distances and phoneme accuracy (2/2) Correlation coefficient 0.97

16. Linguistic characteristics

17. 17 Written text and spontaneous speech corpora Mainichi newspaper(NP)  Written text corpus News commentary(NC)  Transcription of utterances spoken based on prepared text Academic presentations(AP) (in CSJ) Extemporaneous presentations(EP) (in CSJ) Dialogue(D) (in CSJ)

18. 18 Part-of-speech observation frequency The frequency of nouns is much higher in the newspaper corpus than in the spontaneous speech. The frequency of fillers is much higher in the dialogue than in news commentary and presentations. NounFillers

19. 19 Trigrams are built as statistical language models for each speaking style, and test-set perplexity is measured for every combination of the styles. Test-set perplexity for spontaneous speech is roughly five times larger than that for written newspaper texts. Perplexity matrix Diagonal elements

20. 20 Distance matrix for visualization Visualization of relationships between the language models Symmetrization of the perplexity matrix as follows: Symmetri- zation PP matrix (PP(a ij )) Distance matrix (D(d ij )) Visualization

21. 21 Correction Equation (3) in the paper is wrong. Correct equation (3) is as follows:

22. 22 Difference between language models Relationship between the language models projected onto a two-dimensional space derived from the distance matrix using MDS (Multidimensional scaling) method Newspaper text and dialogue are situated at two extreme positions. Presentations and news commentary are situated in between.

23. 23 Relationship between perplexity and word accuracy (1/2) Investigation of relationship between test-set perplexity and word accuracy Acoustic model  A common model for all speaking styles  Trained on the data from 10 males and 10 females for each speaking style (about 750K phoneme samples) Language models  Separate models for each speaking style

24. 24 Relationship between perplexity and word accuracy (2/2) The test-set perplexity (diagonal elements in the PP matrix) and word accuracy Experimental results indicate that they have a high correlation of –0.98 between the test-set perplexity and recognition accuracy across different speaking styles. Correlation coefficient –0.98

25. 25 Conclusion(1/2) Clarified differences of acoustic and linguistic characteristics between spontaneous speech and read speech. Acoustic characteristics  Spectral distribution of spontaneous speech is reduced in comparison with that of read speech.  The more spontaneous, the smaller the distances between phonemes.  There is a high correlation between the mean phoneme distance and the phoneme recognition accuracy.  Spontaneous speech can be characterized by the reduction of spectral space in comparison with that of read speech, and this is one of the major factors contributing to the decrease in phoneme recognition accuracy.

26. 26 Conclusion(2/2) Linguistic characteristics  The perplexity for language models of spontaneous speech is significantly higher than that for written text.  Spontaneous speech frequently includes ungrammatical phenomena and linguistic variations, including repetitions and repairs  There is a high correlation between the test-set perplexity and the word recognition accuracy.  Increment of the test-set perplexity of spontaneous speech is one of the major factors contributing to the decrease in word recognition accuracy

27. 27 Future research Analysis over wider ranges of spontaneous speech using utterances other than those included in the CSJ  Is the relationship between phoneme distances and phoneme recognition accuracy general?  Is the relationship between test-set perplexity and word recognition accuracy general? How to incorporate filled pauses, repairs, hesitations, repetitions, partial words, and disfluencies for spontaneous speech Investigations of how we can use these results obtained in this paper for improving recognition performance of spontaneous speech  Creating methods for adapting acoustic and language models to spontaneous speech

28. Thank you very much for your kind attention! E-mail: masa@furui.cs.titech.ac.jp