Automatic Pronunciation Scoring of Specific Phone Segments for Language Instruction EuroSpeech 1997 Authors: Y. Kim, H. Franco, L. Neumeyer Presenter: Davidson Date: 2009/10/14

2/20 Outline  Introduction  Speech database  Consistency of human ratings  Pronunciation scoring  Experimental results  Conclusions and future work

3/20 Introduction  CAPT (Computer-assisted Pronunciation Training) French spoken by Americans  Pronunciation scoring on Entire sentences Specific phone segments (10 phones)  Number of phone utterances vs. reliable feedback on a speaker ’ s pronunciation proficiency

4/20 Speech database (1/2)  Target language: French  Native corpus 100 Parisian French speakers Used to train models for speech recognition  Nonnative corpus 100 American students Rated by 5 French teachers Only the selected phone segments in each utterance were rated

5/20 Speech database (2/2)  4656 phone segments were selected and rated, consisting of 10 phones /an/, /eh/, /eo/, /eu/, /ey/, /in/, /on/, /r/, /uw/, /uy/  Score scale: 1 (unintelligible) to 5 (native- like)  Serious disfluent or unacceptable audio quality sentences were discarded  Each rater scored some utterances more than once without being informed Self-consistency test

6/20 Consistency of human ratings (1/4)  Inter-rater correlation Phone level Phone-specific speaker level Overall speaker level  Intra-rater correlation  Correlation between and : Vector length Standard deviation

7/20 Consistency of human ratings (2/4)  Phone level inter-rater correlation Speaker 1Speaker 2Speaker 100 Sent 1 Sent P1P Rater1 Rater

8/20 Consistency of human ratings (3/4)  Phone-specific speaker level inter-rater correlation Speaker 1Speaker 2Speaker /an/ Rater1 Rater /uy/ /eh/ 4 5  Overall speaker level inter-rater correlation Speaker Rater1 Rater Speaker 2Speaker 100

9/20 Consistency of human ratings (4/4)  Average inter- and intra-rater correlation across all phones for 5 human raters Corr typeLevel# of scoresCorr. inter Phone Sentence n/a0.65 Phone-specific speaker Overall speaker n/a0.87 intra Phone

10/20 Pronunciation scoring  HMM-based log-likelihood scores  HMM-based log-posterior probability scores  Segment duration scores

11/20 HMM-based log-likelihood scores  For each phone segment, the log-likelihood score is defined as: where = starting frame index of the phone segment = number of frames of the phone segment = observation vector = the i th model = likelihood of the current frame

12/20 HMM-based log-posterior probability scores  For each phone segment, the frame-based posterior probability is defined as: where = prior probability of the phone class  The posterior score for the phone segment is then defined as:

13/20 Segment duration scores  Phone lengths are measured in frames  Phone lengths are normalized by speakers ’ rate-of-speech  Log-probability of the normalized duration is computed using a discrete distribution of durations (trained from native training data)

14/20 Experimental results  Test set: (average) 30 sentences from each of the 100 American speakers  Experiments Human-machine correlation for phone scores Effect of varying the amount of speaker data

15/20 Human-machine correlation for phone scores (1/3)  Phone level correlations with about 450 phone scores in each phone class

16/20 Human-machine correlation for phone scores (2/3)  Phone-specific speaker level correlations with a total of 4656 phone segments across 100 speakers

17/20 Human-machine correlation for phone scores (3/3)  Comparison between human-human and human-machine correlation at the phone level and phone-specific speaker level Human-machine correlation Human-human correlation

18/20 Effect of varying the amount of speaker data (1/2)  To evaluate the system ’ s performance as a function of the number of test utterances per speaker  The number (N) of phone scores per speakers varied from 10 to 320 Phone proportion is preserved  Then, for each N, the speaker level correlation is computed between: Speaker-averaged machine scores (of N scores) Speaker-averaged human scores (of entire human score data)

19/20 Effect of varying the amount of speaker data (2/2) N=40 Posterior Duration Likelihood

20/20 Conclusions and future work  Posterior score performs better than likelihood and duration scores  The system ’ s performance is comparable to human raters at speaker level but not as good as that at phone level  Future work: More human-rated utterances Scoring algorithms with mispronunciation detection