1. Research on the Modeling of Chinese Continuous Speech Recognition
Xiao Xi

2. Content Tri-phone Modeling of Chinese Continuous Speech
Pinyin Pre-processing of Language Model

3. Issue on selecting appropriate acoustic units
Acceptable Accuracy
Unit should be accurate enough to represent acoustic-phonetic events
Easy to model
New word model can be derived from these predefined units
Easy to train
We should have enough data to estimate the unit parameters
Word, syllable, semi-syllable or phoneme unit ?

4. Characteristics of Chinese
Chinese speech is a tonal speech
4 basic tone patterns
The tone is meaningful for understanding. e.g. mai3(买 buying) and mai4(卖 selling) have contrary meaning
About 1254 tonal syllables,408 un-tonal syllables
Pinyin is the transcription of prounciation

5. Characteristics of Chinese
All characters are monosyllabic
Each syllable is composed of an initial and a final semi-syllable
Initial semi-syllable is majorly the consonant of a Chinese syllable
Final semi-syllable follows the initial semi-syllable and is majorly of a simple or compound vowel

6. Bi-phone modeling of Initial-Final structure
Bi-phone modeling only consider the intra syllable constrain, i.e. the literally reasonable combination of initial and final semi-syllable based on phonetic knowledge
100 initial models
41 final models (un-tone) or 164 final models( considering tone )

7. Bi-phone modeling by HMM
An initial is modeled by 2 states HMM
A final is modeled by 4 states HMM

8. Tri-phone modeling of Initial-Final structure
Considering the left-context and right-context of a semi-syllable
Semi-syllables with different context are regarded as different tri-phones
Tri-phone model number is increased dramatically
Sharing techniques is employed to trade off between the model accuracy and the shortage of training corpus

9. Tri-phone modeling by HMM
Considering the co-articulation influence of the previous syllable

10. The Sharing Strategy
Too many models if we evolve tri-phone model from the bi-phone model. e.g,164*100*164 tri-phone
The Intra syllable’s initial-final model remains unchanged
The Inter syllable tri-phone expansion is derived from the final class-initial class definition (sharing)

11. The Sharing Strategy (cont.)
Classification of the final model
Categorized into 29 classes according the ending vowel
30 classes if considering SILENCE
Two schemes
Un-tonal classification, 29 classes
Tonal classification, 112 classes

12. The Sharing Strategy (cont.)
Classification of the initial model
Categorized into 27 classes, considering the influence of the previous FINAL
28 classes if considering SILENCE
The tone of syllable is regarded as less important in initial modeling

13. The Sharing Strategy (cont.)

14. Tri-phone Experiment
Different Tri-phone models for Experiment. ( Bi-phone is the baseline system)

15. Experimental Results – 1st Cand
863 + Intel bj sh male TEST ON 98test data
Experimental Results – 1st Cand

16. Experimental Results – 25 Cands
863 + Intel bj sh male TEST ON 98test data
Experimental Results – 25 Cands

17. Error rate vs. model complexity

18. Advantage of Phonetic Context Based Tri-phone Modeling
Training algorithm is very easy to implement and is time-saving
Less training data is possible
Tri-phone models based on phonetic context knowledge are accurate and can significantly improve the ASR performance

19. Language Model for Chinese Continuous Speech Recognition
Capable of processing multi-length and multi-candidate output of the ASR
Tolerant of deletion errors, insert error and substitute errors of the ASR
Convert Pinyin strings to Chinese characters correctly

20. Framework of speech recognition
where W is the sentence of speech, A is Pinyin, O is the observation of the sentence’s acoustic feature. The sentence W comprises of L Chinese characters

21. Framework of speech recognition (cont.)
For simplicity, substitute Σ calculation by the likely-hood of the best Pinyin candidate, then
Here P(W, A) is Chinese language model, P(O/A) is the acoustic model.
In the following , we will focus on the language model.

22. Multi-pass strategy of Language Model
Here P(W/A) is the Pinyin to Chinese character conversion model, P(A) is the pinyin language model, where P(W) is Chinese character language model. P(O/A) is the acoustic model.
So in the multi-pass language model, the Pinyin model is used to refine the output of acoustic model and then fed into the P(W/A) model

23. Advantage of the multi-pass language model
Pinyin based tri-gram model is much more simplified than the character based tri-gram model.
At most1254 tonal syllables
At least 6000 frequently used Chinese characters
Acceptable time to process multi-length and multi-candidate output from the acoustic model

24. Convert Pinyin Lattice to Words
Example: the speech is “我们来了”（We comes），The PinYin Lattice after rescoring is:
wo3 men2 lai2 le1
wo4 men1 lai4 le4
wo1 men lei1 lo1
wu3 min la lei1
The word graph is created by checking lexicon and the LM (trigram):

25. 我们 来 了 我 们 赖 勒
Start 握 门 莱 乐
End 五 民 拉 垒 屋门 啦 嘞

26. The result is the best way of word graph:我们 来 了 我 们 赖 勒
Start 握 门 莱 乐
End 五 民 拉 垒 屋门 啦 嘞

27. Experiment on Pinyin language model
Training corpus : 20 millions Chinese characters
Testing Sentence: 1680 sentences, about Chinese characters.
Acoustic Model: Tri-phone duration distribution based HMM model

28. Experiment result

29. Conclusion from Experiment on Pinyin language model
Dramatically improvement on the accuracy of refined candidates
45% improvement for the first candidate’s hit rate by using tri-gram Pinyin model.
Top 20 candidates’ hit rate (97.21%) has exceeded the top 100 candidates’ hit rate(97.12%) of the baseline system

30. The End

31. Q & A