1. Statistical Language Modeling for Speech Recognition and Information Retrieval
Berlin Chen
Department of Computer Science & Information Engineering
National Taiwan Normal University

2. Outline What is Statistical Language Modeling
Statistical Language Modeling for Speech Recognition, Information Retrieval, and Document Summarization
Categorization of Statistical Language Models
Main Issues for Statistical Language Models
Conclusions

3. What is Statistical Language Modeling ?
Statistical language modeling (LM), which aims to capture the regularities in human natural language and quantify the acceptance of a given word sequence
Adopted from Joshua Goodman’s public presentation file

4. What is Statistical LM Used for ?
It has continuously been a focus of active research in the speech and language community over the past three decades
It also has been introduced to the information retrieval (IR) problems, and provided an effective and theoretically attractive probabilistic framework for building IR systems
Other application domains
Machine translation
Input method editor (IME)
Optical character recognition (OCR)
Bioinformatics
etc.

5. Statistical LM for Speech Recognition
Speech recognition: finding a word sequence out of the many possible word sequences that has the maximum posterior probability given the input speech utterance
Acoustic Modeling
Language Modeling

6. Statistical LM for Information Retrieval
Information retrieval (IR): identifying information items or “documents" within a large collection that best match (are most relevant to) a “query” provided by a user that describes the user’s information need
Query-likelihood retrieval model: a query is considered generated from an “relevant” document that satisfies the information need
Estimate the likelihood of each document in the collection being the relevant document and rank them accordingly
Query Likelihood
Document Prior Probability

7. Statistical LM for Document Summarization
Estimate the likelihood of each sentence of the document being in the summary and rank them accordingly
The sentence generative probability can be taken as a relevance measure between the document and sentence
The sentence prior probability is, to some extent, a measure of the importance of the sentence itself
Sentence Generative Probability
(or Document Likelihood)
Sentence Prior Probability

8. n-Gram Language Models (1/3)
For a word sequence , can be decomposed into a product of conditional probabilities
E.g.,
However, it’s impossible to estimate and store if is large (the curse of dimensionality)
multiplication rule
History of wi

9. n-Gram Language Models (2/3)
n-gram approximation
Also called (n-1)-order Markov modeling
The most prevailing language model
E.g., trigram modeling
How do we find probabilities? (maximum likelihood estimation)
Get real text, and start counting (empirically)
History of length n-1
Probability may be zero
count

10. n-Gram Language Models (3/3)
Minimum Word Error (MWE) Discriminative Training
Given a training set of observation sequences , the MWE criterion aims to minimize the expected word errors of these observation sequences using the following objective function
MWE objective function can be optimized with respect to the language model probabilities using Extended Baum-Welch (EBW) algorithm

11. n-Gram-Based Retrieval Model (1/2)
Each document is a probabilistic generative model consisting of a set of n-gram distributions for predicting the query
Document models can be optimized by the expectation-maximization (EM) or minimum classification error (MCE) training algorithms, given a set of query and relevant document pairs
Features:
1. A formal mathematic framework
2. Use collection statistics but not heuristics
3. The retrieval system can be gradually improved through usage
Query

12. n-Gram-Based Retrieval Model (2/2)
MCE training
Given a query and a desired relevant doc , define the classification error function as:
“>0”: means misclassified; “<=0”: means a correct decision
Transform the error function to the loss function
Iteratively update the weighting parameters, e.g.,
Also can take all irrelevant doc
in the answer set into consideration

13. n-Gram-Based Summarization Model (1/2)
Each sentence of the spoken document is treated as a probabilistic generative model of n-grams, while the spoken document is the observation
: the sentence model, estimated from the sentence
: the collection model, estimated from a large corpus
In order to have some probability of every word in the vocabulary

14. n-Gram-Based Summarization Model (2/2)
Relevance Model (RM)
In order to improve the estimation of sentence models
Each sentence has its own associated relevance model , constructed by the subset of documents in the collection that are relevant to the sentence
The relevance model is then linearly combined with the original sentence model to form a more accurate sentence model

15. Categorization of Statistical Language Models (1/4)

16. Categorization of Statistical Language Models (2/4)
1. Word-based LMs
The n-gram model is usually the basic model of this category
Many other models of this category are designed to overcome the major drawback of n-gram models
That is, to capture long-distance word dependence information without increasing the model complexity rapidly
E.g., mixed-order Markov model and trigger-based language model, etc.
2. Word class (or topic)-based LMs
These models are similar to the n-gram model, but the relationship among words is constructed via (latent) word classes
When the relationship is established, the probability of a decoded word given the history words can be easily found out
E.g., class-based n-gram model, aggregate Markov model and word topical mixture model (WTMM)

17. Categorization of Statistical Language Models (3/4)
3. Structure-based LMs
Due to the constraints of grammars, rules for a sentence may be derived and represented as a parse tree
Then, we can select among candidate words by the sentence patterns or head words of the history
E.g., structured language model
4. Document class (or topic)-based LMs
Words are aggregated in a document to represent some topics (or concepts). During speech recognition, the history is considered as an incomplete document and the associated latent topic distributions can be discovered on the fly
The decoded words related to most of the topics that the history probably belongs to can be therefore selected
E.g., mixture-based language model, probabilistic latent semantic analysis (PLSA) and latent Dirichlet allocation (LDA)

18. Categorization of Statistical Language Models (4/4)
Ironically, the most successful statistical language modeling techniques use very little knowledge of what language is
may be a sequence of arbitrary symbols, with no deep structure, intention, or thought behind them
F. Jelinek said “put language back into language modeling”
“Closing remarks” presented at the 1995 Language Modeling Summer Workshop, Baltimore

19. Main Issues for Statistical Language Models
Evaluation
How can you tell a good language model from a bad one
Run a speech recognizer or adopt other statistical measurements
Smoothing
Deal with data sparseness of real training data
Various approaches have been proposed
Adaptation
The subject matters and lexical characteristics for the linguistic contents of utterances or documents (e.g., news articles) might be are very diverse and are often changing with time
LMs should be adapted consequently
Caching: If you say something, you are likely to say it again later
Adjust word frequencies observed in the current conversation

20. Evaluation (1/7)
Two most common metrics for evaluation a language model
Word Recognition Error Rate (WER)
Perplexity (PP)
Word Recognition Error Rate
Requires the participation of a speech recognition system (slow!)
Need to deal with the combination of acoustic probabilities and language model probabilities (penalizing or weighting between them)

21. Evaluation (2/7) Perplexity
Perplexity is geometric average inverse language model probability (measure language model difficulty, not acoustic difficulty/confusability)
Can be roughly interpreted as the geometric mean of the branching factor of the text when presented to the language model
For trigram modeling:

22. Evaluation (3/7) More about Perplexity
Perplexity is an indication of the complexity of the language if we have an accurate estimate of
A language with higher perplexity means that the number of words branching from a previous word is larger on average
A langue model with perplexity L has roughly the same difficulty as another language model in which every word can be followed by L different words with equal probabilities
Examples:
Ask a speech recognizer to recognize digits: “0, 1, 2, 3, 4, 5, 6, 7, 8, 9” – easy – perplexity 10
Ask a speech recognizer to recognize names at a large institute (10,000 persons) – hard – perplexity  10,000

23. Evaluation (4/7) More about Perplexity (Cont.)
Training-set perplexity: measures how the language model fits the training data
Test-set perplexity: evaluates the generalization capability of the language model
When we say perplexity, we mean “test-set perplexity”

24. Evaluation (5/7) Is a language model with lower perplexity is better?
The true (optimal) model for data has the lowest possible perplexity
The lower the perplexity, the closer we are to the true model
Typically, perplexity correlates well with speech recognition word error rate
Correlates better when both models are trained on same data
Doesn’t correlate well when training data changes
The 20,000-word continuous speech recognition for Wall Street Journal (WSJ) task has a perplexity about 128 ~ 176 (trigram)
The 2,000-word conversational Air Travel Information System (ATIS) task has a perplexity less than 20

25. Evaluation (6/7)
The perplexity of bigram with different vocabulary size

26. Evaluation (7/7) A rough rule of thumb (recommended by Rosenfeld)
Reduction of 5% in perplexity is usually not practically significant
A 10% ~ 20% reduction is noteworthy, and usually translates into some improvement in application performance
A perplexity improvement of 30% or more over a good baseline is quite significant
Vocabulary
Perplexity
WER
zero |one |two |three |four
|five |six |seven |eight |nine 10 5
John |tom |sam |bon |ron |
|susan |sharon |carol |laura |sarah 7
bit |bite |boot |bait |bat
|bet |beat |boat |burt |bart 9
Perplexity cannot always
reflect the difficulty of a
speech recognition task
Perplexity cannot always reflect the difficulty of a speech recognition task
Tasks of recognizing 10 isolated-words using IBM ViaVoice

27. Smoothing (1/3)
Maximum likelihood (ML) estimate of language models has been shown previously, e.g.:
Trigam probabilities
Bigram probabilities
count

28. Smoothing (2/3) Data Sparseness
Many actually possible events (word successions) in the test set may not be well observed in the training set/data
E.g. bigram modeling
P(read|Mulan)= P(Mulan read a book)=0
P(W)= P(X|W)P(W)=0
Whenever a string such that occurs during speech recognition task, an error will be made

29. Smoothing (3/3) Smoothing
Assign all strings (or events/word successions) a nonzero probability if they never occur in the training data
Tend to make distributions flatter by adjusting lower probabilities upward and high probabilities downward

30. Smoothing: Simple Models
Add-one smoothing
For example, pretend each trigram occurs once more than it actually does
Add delta smoothing
Work badly! DO NOT DO THESE TWO.

31. Smoothing: Back-Off Models
The general form for n-gram back-off
: normalizing/scaling factor chosen to make the conditional probability sum to 1
I.e.,
P(蔣介石 | 中華民國 總統)=0.3
P(嚴家淦 | 中華民國 總統)=0.1
P(蔣經國 | 中華民國 總統)=0.3
P(李登輝 | 中華民國 總統)=0.2
P(陳水扁 | 中華民國 總統)=0, but P(陳水扁 | 總統)=0.3
P(蘇貞昌 | 中華民國 總統)=0, but P(蘇貞昌 | 總統)=0.1 ……
n-gram
smoothed (n-1)-gram

32. Smoothing: Interpolated Models
The general form for Interpolated n-gram back-off
The key difference between backoff and interpolated models
For n-grams with nonzero counts, interpolated models use information from lower-order distributions while back-off models do not
Moreover, in interpolated models, n-grams with the same counts can have different probability estimates
count

33. Caching (1/2)
The basic idea of cashing is to accumulate n-grams dictated so far in the current document/conversation and use these to create dynamic n-grams model
Trigram interpolated with unigram cache
Trigram interpolated with bigram cache

34. Caching (2/2) Real Life of Caching
Someone says “I swear to tell the truth”
System hears “I swerve to smell the soup”
Someone says “The whole truth”, and, with cache, system hears “The toll booth.” – errors are locked in
Caching works well when users correct as they go, poorly or even hurts without corrections
Cache remembers!
Swerve:突然轉向;轉彎;偏離方向
Adopted from Joshua Goodman’s public presentation file

35. LM Integrated into Speech Recognition
Theoretically,
Practically, language model is a better predictor while acoustic probabilities aren’t “real” probabilities
Penalize insertions
E.g.,

36. n-Gram Language Model Adaptation (1/4)
Count Merging
n-gram conditional probabilities form a a multinominal distribution
The parameters form sets of independent Dirichlet distributions with hyperparameters
The MAP estimate is the posterior distribution of
All possible N-gram histories
Vocabulary Size

37. n-Gram Language Model Adaptation (2/4)
Count Merging (cont.)
Maximize the posterior distribution of w.r.t. the constraint
Differentiate w.r.t.
Largrange Multiplier

38. n-Gram Language Model Adaptation (3/4)
Count Merging (cont.)
Parameterization of the prior distribution (I):
The adaptation formula for Count Merging
E.g.,
Background Corpus
Adaptation Corpus

39. n-Gram Language Model Adaptation (4/4)
Model Interpolation
Parameterization of the prior distribution (II):
The adaptation formula for Model Interpolation
E.g.,

40. Known Weakness in Current n-Gram LM
Brittleness Across Domain
Current language models are extremely sensitive to changes in the style or topic of the text on which they are trained
E.g., conversations vs. news broadcasts, fictions vs. politics
Language model adaptation
In-domain or contemporary text corpora/speech transcripts
Static or dynamic adaptation
Local contextual (n-gram) or global semantic/topical information
False Independence Assumption
In order to remain trainable, the n-gram modeling assumes the probability of next word in a sentence depends only on the identity of last n-1 words
n-1-order Markov modeling

41. Conclusions
Statistical language modeling has demonstrated to be an effective probabilistic framework for NLP, ASR, and IR-related applications
There remains many issues to be solved for statistical language modeling, e.g.,
Unknown word (or spoken term) detection
Discriminative training of language models
Adaptation of language models across different domains and genres
Fusion of various (or different levels of) features for language modeling
Positional Information?
Rhetorical (structural) Information?

42. References
J. R. Bellegarda. Statistical language model adaptation: review and perspectives. Speech Communication 42(11), , 2004
X., W. Liu, B. Croft. Statistical language modeling for information retrieval. Annual Review of Information Science and Technology 39, Chapter 1, 2005
R. Rosenfeld. Two decades of statistical language modeling: where do we go from here? Proceedings of IEEE, August 2000
J. Goodman, A bit of progress in language modeling, extended version. Microsoft Research Technical Report MSR-TR , 2001
H.S. Chiu, B. Chen. Word topical mixture models for dynamic language model adaptation. ICASSP2007
J.W. Kuo, B. Chen. Minimum word error based discriminative training of language models. Interspeech2005
B. Chen, H.M. Wang, L.S. Lee. Spoken document retrieval and summarization. Advances in Chinese Spoken Language Processing, Chapter 13, 2006

43. Maximum Likelihood Estimate (MLE) for n-Grams (1/2)
Given a a training corpus T and the language model
Essentially, the distribution of the sample counts with the same history referred as a multinominal (polynominal) distribution
N-grams with
same history
are collected
together
…陳水扁 總統 訪問 美國 紐約 … 陳水扁 總統 在 巴拿馬 表示 …
P(總統|陳水扁)=?

44. Maximum Likelihood Estimate (MLE) for n-Grams (2/2)
Take logarithm of , we have
For any pair , try to maximize and subject to