TBALL ASR Work Summary USC group: Joseph Tepperman, Matt Black, Abe Kazemzadeh, Matteo Gerosa, Sungbok Lee, Shri Narayanan

ASR contributions of this year Disfluency detection – Will be presented at InterSpeech 2007 Letter name and letter sound verification Bayesian Network approach for pronunciation scoring – Will be presented at InterSpeech 2007

Disfluency Evaluation Figure 1: which types of disfluencies are rated more or less fluent. Table 2: what is more important, fluency or accuracy.

Disfluency Detection Figure 2: grammar. Figure 3: classification algorithm. Table 3: Results for different disfluency types.(9% false alarm)‏

Letter name and letter sound verification Letter Name verification – Using word (letter) level models – 82% recognition accuracy, ~90% verification. – Need new performance measurement with new/noisy data letter sound verification – Baseline of using regular monophone models still outperforms experimental techniques. – New approach: to segment audio as a preprocessing step to deal with repetitions.

Bayesian Network approach for pronunciation scoring

Classifier performance over different feature sets

Bayes Nets Results

Question and Answering Evaluation Interesting task b/c a child may answer in different ways (specifically or generally, making up an answer out of the blue). Also, the task can be thought of as the bridge between mechanical reading and comprehension. Difficulty getting people to do the evaluation. – We had to resort to a gimmick….

The Tball Hall of Fame: Thanks to those who completed the evaluation! Joe Tepperman Jan Powell Matt Black Abe Kazemzadeh Patti Price Mia Callahan Christy Boscardin

Evaluation Results Out of possible pairings between evaluators, 8000 agreed and 3136 didn't. – Approx. 72% agreement. Still much analysis to do. Do the evaluation... it's fun.

SQLite This is a handy database tool that I found out about recently – I used it for the web evaluation for question answering. – It’s used in a number of well known programs (e.g., preferences in firefox, google desktop widgets)‏ I thought it might be ideal for the db on the laptops. – No installation, just a executable (or imported libraries)‏ – Command line or gui usage. – Bindings for common languages using DBI interface.

Discussion & Further studies Next steps: – Analyzing the outcomes of the case study. – Improve robustness of the existing algorithms for verification and disfluency detection. – Further improvement of the current automatic scoring method. – Making use of all the data we have collected for analysis and improvements including "comprehension" of spontaneous speech (i.e., open-ended question evaluation)