Smoothing Issues in the Strucutred Language Model The Center for Language and Speech Processing The Johns Hopkins University 3400 N. Charles Street, Barton Hall Baltimore, MD 21218 Woosung Kim, Sanjeev Khudanpur, and Jun Wu The Center for Language and Speech Processing, The Johns Hopkins University {woosung, sanjeev, junwu}@clsp.jhu.edu Introduction The Structured Language Model(SLM) - An attempt to exploit the syntactic structure of natural language - Consists of a predictor, a tagger and a parser - Jointly assigns a probability to a word sequence and parse structure - Still suffers from data sparseness problem, Deleted Interpolation(DI) has been used  Use of Kneser-Ney smoothing to improve the performance Experiment Result N-Best Rescoring Test Set PPL as a Function of l ASR WER for SWB Two corpora Wall Street Journal(WSJ) Upenn Treebank for LM PPL test Switchboard(SWB) For ASR WER test as well as LM PPL Tokenization Original SWB tokenization Examples : They’re, It’s, etc.  Not Suitable for syntactic analysis Treebank tokenization Examples : They ‘re, It ‘s, etc. Kneser-Ney Smoothing Backoff Nonlinear Interpolation Smoothing 3gram SLM Inptl Deleted Intpl 39.1% 38.6% 38.2% KN-BO(Predictor) 38.3% 37.7% 37.5% KN-BO(All Modules) 37.8% Nonlinear Intpl 38.1% 37.6% NI w/Deleted Est. Speech Recognizer (Baseline LM) 100 Best Hyp Speech (New LM) Rescoring 1 hypothesis Database Size Specifications(in Words) Item WSJ SWB Word Voc. Part-Of-Speech Tags Non-Terminal Tags Parser Operations 10K(open) 40 54 136 21K(closed) 49 64 112 LM Dev. Set LM Check Set LM Test Set ASR Test Set 885K 117K 82K - 2.07M 216K 20K Concluding Remarks KN smoothing of the SLM shows modest but consistent improvements – both PPL and WER Future Work SLM with Maximum Entropy Models But Maximum Entropy Model training requires heavy computation Fruitful results in the selection of features for the Maximum Entropy Models The Structured Language Model(SLM) Example of a Partial Parse Probability estimation in the SLM ended VP with PP loss NP of PP Language Model Perplexity contract NP loss NP cents NP WSJ Corpus SWB EM Iter. 3gram SLM Intpl Smoothing EM0 EM3 70 73 72 67 66 Deleted 162 166 154 149 146 64 63 60 KN-BO (Predictor) 152 139 137 All Modules 170 153 141 140 65 61 Nonlinear 132 131 NI w/Deleted Estimation 145 150 130 The contract ended with a loss of 7 cents after DT NN VBD IN DT NN IN CD NNS Parse tree probability predictor tagger LM PPL parser This research was partially supported by the U.S. National Science Foundation via STIMULATE grant No. 9618874