1. 1 Evaluating the Effect of Predicting Oral Reading Miscues Satanjeev Banerjee, Joseph Beck, Jack Mostow Project LISTEN (www.cs.cmu.edu/~listen) Carnegie Mellon University Funding: NSF IERI

2. 2 Why Predict Miscues? Reading Tutor helps children learn to read. Speech recognizer listens for miscues (reading errors) –E.g.: listen for “hat” if sentence to be read has word “hate” Accurate miscue prediction helps miscue detection.

3. 3 Real Word Substitutions Miscues = substitutions, omissions, insertions Real word substitution = misread target word as another word –E.g. read “hat” instead of “hate” Most miscues are real word substitutions ICSLP-02: predicted real word substitutions Here: evaluate effect on substitution detection

4. 4 How Evaluate Substitution Detection? What child saidIhathavingmilkeverya Correct textIhatehavingmilkeveryday What ASR heard Ihatehavingmilleveryhate substitution undetectedfalse alarmsubstitution detected substitution 1 2 Substitution detection rate = # substitutions detected # substitutions child made = 1 4 False alarm rate = # false alarms # words correctly read =

5. 5 Evaluation Data Sentences read by 25 children aged 6 to 10 Correctly readIncorrectly read Content tokens 5,981335 Function tokens 3,166147

6. 6 Rote Method Uses the University of Colorado miscue database. For each target word –Sort substitutions by # children who made them. –Predict that the top n substitutions will reoccur, for this word. Model type Substitution detection rate False alarm rate No predicted substitutions 21.58 %2.42 % Top 122.82 %2.88 % Top 224.90 %3.27 %

7. 7 Extrapolative Method Predict the probability that a word is a likely substitution for another word –Pr ( substitution “hat” | target “hate”) Use machine learning to induce a classifier Train using University of Colorado miscue database. Some features (more in paper) Candidate substitution Target word Spelling edit distance = 1H.A.T.EH.A.T Phonetic distance = 1/HH EY T//HH AE T/ Rank in descending sorted frequency table 13641887

8. 8 Extrapolative Method cont’d Given a target word, predict substitution if Pr ( substitution candidate | target word ) > threshold Model type Substitution detection rate False alarm rate No predicted substitutions 21.58 %2.42 % Pr >= 0.9923.03 %2.77 % Pr >= 0.9524.48 %3.67 %

9. 9 Combining Rote and Extrapolative Aim: Get n substitutions for a given word. Step 1: Use top n substitutions from rote. Step 2: If rote predicts k substitutions, k < n, –Then add top n – k substitutions from extrapolative. Intuition: rote is more accurate, so use when available. If not available, fall back on extrapolative.

10. 10 Results from Combining Algorithms Model type Substitution detection rate False alarm rate Top 125.73 %3.77 % Truncation24.69 %4.29 % Top 231.54 %4.62 % Theoretical max 42.53 %2.90 % Truncation = The first 2 to n-2 phonemes of a word – models false starts. [/K AE/ and /K AE N/ for /K AE N D IY/; none for “hate”] Theoretical max = use only those miscues the child actually made.

11. 11 Conclusion Evaluated effect on substitution detection of –Two previously published algorithms –A combination of the two algorithms. Combined approach improved on current configuration (truncations) by –Reducing false alarms by 0.52% abs (12% rel) –Increasing miscue detection by 1.04% (4.2% rel) Take-home sound byte: Listening for specific reading mistakes can help detect them!