Network Training for Continuous Speech Recognition Author: Issac John Alphonso Inst. for Signal and Info. Processing Dept. Electrical and Computer Eng. Mississippi State University Contact Information: Box 0452 Mississippi State University Mississippi State, Mississippi Tel: Fax: URL: isip.msstate.edu/publications/books/msstate_theses/2003/network_training/

INTRODUCTION ABSTRACT A traditional trainer uses an expectation maximization (EM) based supervised training framework to estimate the parameters of a speech recognition system. EM- based parameter estimation for speech recognition is performed using several complicated stages of iterative re-estimation. These stages are prone to human error. This thesis describes a new network training paradigm that reduces the complexity of the training process, while retaining the robustness of the EM-based supervised training framework. The network trainer can achieve comparable recognition performance to a traditional trainer while alleviating the need for complicated systems and training recipes for speech recognition systems.

INTRODUCTION ORGANIZATION Motivation: Why do we need a new training paradigm? Theoretical: Review the EM-based supervised training framework. Network Training: The differences between the network training and traditional training. Experiments: Verification of the approach using industry standard databases (e.g., TIDigits, Alphadigits and Resource Management). Motivation Network Training Theoretical Background Experiments Conclusion & Future Work

INTRODUCTION MOTIVATION A traditional trainer uses an EM-based framework to estimate the parameters of a speech recognition system. EM-based parameter estimation is performed in several complicated stages which are prone to human error. A network trainer reduces the complexity of the training process by employing a soft decision criterion. A network trainer achieves comparable performance and retains the robustness of the EM-based framework.

THEORETICAL BACKGROUND COMMUNICATION THEORETIC APPROACH Message Source Linguistic Channel Articulatory Channel Acoustic Channel Observable: MessageWordsSounds Features Maximum likelihood formulation for speech recognition: P(W|A) = P(A|W) P(W) / P(A) Objective: minimize the word error rate Approach: maximize P(W|A) during training Components: P(A|W) : acoustic model (HMM’s/GMM’s) P(W) : language model (statistical, FSN’s, etc.)

THEORETICAL BACKGROUND MAXIMUM LIKELIHOOD The approach treats the parameters of the model as fixed quantities whose values need to be estimated. The model parameters are estimated by maximizing the log likelihood of observing the training data. The estimation of the parameters is computationally tractable due to the availability of efficient algorithms.    T t i oPOP 1 )]|(log[)]|(log[

THEORETICAL BACKGROUND EXPECTATION MAXIMIZATION A general framework that can be used to determine the maximum likelihood estimates of the model parameters. The algorithm iteratively estimates the likelihood of the model by maximizing Baum’s auxiliary function. The expectation maximization algorithm is guaranteed to converge to the maximum likelihood estimate.   q qOqOPQ)|,log()|,(),(

THEORETICAL BACKGROUND HIDDEN MARKOV MODELS A random process that consists of a set of states and their corresponding transition probabilities: The priori probabilities: The state transition probabilities: The state emission probabilities: ),0(jstatetP j  ),|,1(i ttimejstatettimePa ij  ),|()(jstatettimeOPOb tj 

NETWORK TRAINER TRAINING RECIPE The flat start stage segments the acoustic signal and seed the speech and non-speech models. The context-independent stage inserts and optional silence model between words. The state-tying stage clusters the model parameters via linguistic rules to compensate for sparse training data. The context-dependent stage is similar to the context- independent stage (words are modeled using context). Flat Start CI Training State Tying CD Training Context-Independent Context-Dependent

NETWORK TRAINER TRANSCRIPTIONS sil hh v v Traditional Trainer: ae sil SILENCE HAVE SILENCE Network Trainer: The network trainer uses word level transcriptions which does not impose restrictions on the word pronunciation. The traditional trainer uses phone level transcriptions which uses the canonical pronunciation of the word. Using orthographic transcriptions removes the need for directly dealing with phonetic contexts during training.

NETWORK TRAINER SILENCE MODELS Multi-Path: Single-Path: The multi-path silence model is used between words. The single-path silence model is used at utterance ends.

NETWORK TRAINER DURATION MODELING The network trainer uses a silence word which precludes the need for inserting it into the phonetic pronunciation. The traditional trainer deals with silence between words by explicitly specifying it in the phonetic pronunciation. Network Trainer: Traditional Trainer:

NETWORK TRAINER PRONUNCIATION MODELING A pronunciation network precludes the need to use a single canonical pronunciation for each word. The pronunciation network has the added advantage of being able to generalize to unseen pronunciations. Network Trainer: Traditional Trainer:

NETWORK TRAINER OPTIONAL SILENCE MODELING The network trainer uses a fixed silence at utterance bounds and an optional silence between words. We use a fixed silence at utterance bounds to avoid an underestimated silence model.

NETWORK TRAINER SILENCE DURATION MODELING Network training uses a single-path silence at utterance bounds and a multi-path silence between words. We use a single-path silence at utterance bounds to avoid uncertainty in modeling silence.

EXPERIMENTS SPEECH DATABASES 0% 10% 30% 40% 20% Word Error Rate Level Of Difficulty Digits Continuous Digits Command and Control Letters and Numbers Broadcast News Read Speech Conversational Speech

EXPERIMENTS TIDIGITS DATABASE Collected by Texas Instruments in 1983 to establish a common baseline for connected digit recognition tasks. Includes digits from ‘zero’ through ‘nine’ and ‘oh’ (an alternative pronunciation for ‘zero’). The corpora consists of 326 speakers (111, men, 114 women and 101 children).

EXPERIMENTS TIDIGITS: WER COMPARISON StageWERInsertion Rate Deletion Rate Substitution Rate Traditional Trainer 7.7%0.1%2.5%5.0% Network Trainer 7.6%0.1%2.4%5.0% The network trainer achieves comparable performance to the traditional trainer. The network trainer converges in word error rate to the traditional trainer.

EXPERIMENTS TIDIGITS: LIKELIHOOD COMPARISON Iterations Average Log Likelihood _ _ _ _ Network Trainer ______ Traditional Trainer

Collected by the Oregon Graduate Institute (OGI) using the CLSU T1 data collection system. Includes letters (‘a’ through ‘z’) and numbers (‘zero’ through ‘nine’ and ‘oh’). The database consists of 2,983 speakers (1,419 men, 1,533 women and 30 children). EXPERIMENTS ALPHADIGITS (AD) DATABASE

EXPERIMENTS AD: WER COMPARISON The network trainer achieves comparable performance to the traditional trainer. The network trainer converges in word error rate to the traditional trainer. StageWERInsertion Rate Deletion Rate Substitution Rate Traditional Trainer 38.0%0.8%3.0%34.2% Network Trainer 35.3%0.8%2.2%34.2%

EXPERIMENTS AD: LIKELIHOOD COMPARISON Average Log Likelihood Iterations _ _ _ _ Network Trainer ______ Traditional Trainer

Was collected by the Defense Advanced Research Project Agency (DARPA). Includes a collection of spoken sentences pertaining to a naval RM task. The database consists of 80 speakers, each reading two ‘dialect’ sentences plus 40 sentences from the RM text corpus. EXPERIMENTS RESOURCE MANAGEMENT (RM) DATABASE

EXPERIMENTS RM: WER COMPARISON The network trainer achieves comparable performance to the traditional trainer. It is important to note that the 1.8% degradation in performance is not significant (MAPSSWE test). StageWERInsertion Rate Deletion Rate Substitution Rate Traditional Trainer 25.7%1.9%6.7%17.1% Network Trainer 27.5%2.6%7.1%17.9%

EXPERIMENTS RM: LIKELIHOOD COMPARISON Average Log Likelihood Iterations _ _ _ _ Network Trainer ______ Traditional Trainer

Explored the effectiveness of a novel training recipe in the reestimation process of for speech processing. Analyzed performance on three databases. For TIDigits, at 7.6% WER, the performance of the network trainer was better by about 0.1%. For OGI Alphadigits, at 35.3% WER, the performance of the network trainer was better by about 2.7%. For Resource Management, at 27.5% WER, the performance degraded by about 1.8% (not significant). CONCLUSIONS SUMMARY

The results presented use single-mixture context- dependent models for training and recognition. A efficient tree-based decoder is currently under development and context-dependent results are planned. The databases presented all use single pronunciations for each word in the lexicon. The ability to run large databases like Switchboard, which has multiple pronunciations, requires a tree-based decoder. CONCLUSIONS FUTURE WORK

PROGRAM OF STUDY Course No.TitleSemester CS 8990Probabilistic Expert SystemsSpring 2000 ST 8253Linear RegressionFall 2000 ECE 8990Pattern RecognitionSpring 2001 ECE 8990Information TheorySpring 2001 CS 8990Reinforcement LearningFall 2001 CS 8663Neural ComputingFall 2001 ECE 8990Random Signals and SystemsFall 2001 ECE 8990Fundamentals of Speech RecognitionSpring 2002 ECE 8000Research/Thesis APPENDIX PROGRAM OF STUDY

I would like to thank Dr. Joe Picone for his mentoring and guidance through out my graduate program. I would also like to thank Jon Hamaker for his valuable suggestions throughout my thesis. Finally, I would like to thank my co-workers at the Institute for Signal and Information Processing (ISIP) for all their help. APPENDIX ACKNOWLEDGEMENTS

S. Pinker, The Language Instinct, Harper Collins, New York City, New York, USA, L. Rabiner, B. Juang, Fundamentals of Speech Recognition, Prentice Hall, Upper Saddle River, New Jersey, USA, R. O. Duda, P. E. Hart, D. G. Stork, Pattern Classification, John Wiley & Sons, New York City, New York, USA, X. Huang, A. Acero, H. Hon, Spoken Language Processing, Prentice Hall, Upper Saddle River, New Jersey, USA, APPENDIX REFERENCES