1 Less is More? Yi Wu Advisor: Alex Rudnicky

2 People: There is no data like more data!

3 Goal: Use less to Perform more Identifying an informative subset from a large corpus for Acoustic Model (AM) training. Expectation of the Selected Set  Good in Performance  Fast in Selection

4 Motivation The improvement of system will become increasingly smaller when we keep adding data. Training acoustic model is time consuming. We need some guidance on what is the most needed data.

5 Approach Overview Applied to well-transcribed data Selection based on transcription Choose subset that have “uniform” distribution on speech unit (word, phoneme, character)

6 How to sample data wisely? --A simple example k Gaussian distribution with known priorω i and unknown density function f i (μ i,σ i )

7 How to sample wisely? --A simplified example We are given access to at most N examples. We have right to choose how much we want from each class. We train the model use MLE estimator. When a new sample generated, we use our model to determine its class. Question: How to sample to achieve minimum error?

8 The optimal Bayes Classifier If we have the exact form of f i (x), above classification is optimal.

9 To approximate the optimal We use our MLE The true error would be bounded by optimal Bayes error plus error bound for our worst estimated

10 Sample Uniformly We want to sample each class equally.  The data selected will have good coverage on each class.  This will give robust estimation on each class.

11 The Real ASR system

12 Data Selection for ASR System The prior has been estimated independently by language model. To make acoustic model accurate, we want to sample the W uniformly. We can take the unit to be phoneme, character, word. We want their distribution to be uniform.

13 Entropy: Measure for “uniformness” Use the entropy of the word (phoneme) as ways of evaluation  Suppose the word (phoneme) has a sample distribution p 1, p 2 …. p n  Choose subset have maximum -p 1 *log(p 1 )- p 2 *log(p 2 )-... p n *log(p n )) Entropy actually is the KL distance from uniform distribution

14 Computational Issue It is computational intractable to find the transcription set that maximizes the entropy Forward Greedy Search

15 Combination There are multiple entropies we want to maximize. Combination Method  Weighted Sum  Add sequentially

16 Experiment Setup System: Sphinx III Feature: 39 dimension MFCC Training Corpus: Chinese BN 97(30hr)+ GaleY1(810hr data) Test Set: RT04(60 min)

17 Experiment 1 ( use word distribution) Time (hour) Random (all) Max-entropy Table 1

18 More Result 30 h50 h100 h150 h840 h random(all) cctv(bn) ntdv(bn) rfa(bc) bc/bn(ratio)15.4/ / / / /40 9 max- entropy(all) cctv(bn) ntdtv(bn) rfa(bc) bc/bn(ratio)11.0/ / /49.8

19 Experiment 2 (add sequentially with phoneme and character 150hr) CCTVNTDTVRFAALL Random(150h) Max-entropy (word+char) Max-entropy (word+phone) All data (840 hrs) Table 2

20 Experiment 1,2

21 Experiment 3 (with VTLN) CCTVNTDTVRFAALL 150 hr (word+phone) With VTLN Table 3

22 Summary Choose data uniformly according to speech unit Maximize entropy using greedy algorithm Add data sequentially Future Work Combine Multiple Sources Select Un-transcribed Data