1. Professor Junghoo “John” Cho UCLA
Language Models
Professor Junghoo “John” Cho
UCLA

2. Today’s Topics Higher-level vector embedding
Quoc V. Le and Tomas Mikolov: Distributed Representations of Sentences and Documents
Building better language models
Joshua Goodman: A bit of progress in language modeling, MSR Technical Report

3. Paragraph Embedding [Le & Mikolov 2014]
Question: Can we create a vector representation for a phrase, sentence, paragraph, document, etc?
More formally
Convert a sequence of words 𝑤 1 𝑤 2 … 𝑤 𝑛 , into a 𝑘-dimensional vector where “similar sequences” are mapped into similar vector
Q: How?

4. Paragraph Embedding [Le & Mikolov 2014]
Q: How did we do it for words?
A: For any word pair 𝑤 𝑖 , 𝑤 𝑗 find their vector embedding 𝑣 𝑖 , 𝑣 𝑗 such that 𝑃 𝑤 𝑗 𝑤 𝑖 =softmax 𝑣 𝑗 𝑇 𝑣 𝑖
A bias term 𝑏 𝑗 may be added inside softmax
Q: Can we formulate the paragraph embedding problem similarly?

5. PV-DBOW (Bag of Words of Paragraph Vector)
Given 𝑝 𝑖 = 𝑤 1 𝑤 2 … 𝑤 𝑛 , find their vector embedding 𝑞 𝑖 , 𝑣 1 , 𝑣 2 ,…, 𝑣 𝑛 such that 𝑃 𝑤 𝑗 𝑝 𝑖 =softmax 𝑣 𝑗 𝑇 𝑞 𝑖
Again, a bias term 𝑏 𝑗 maybe added
Train 𝑞 𝑖 , 𝑣 𝑗 vectors on large text corpus to maximize the log likelihood 𝑖,𝑗 ∈𝐷 log 𝑃( 𝑤 𝑖 | 𝑝 𝑗 )
𝑞 𝑖
( 𝑣 𝑗 , 𝑏 𝑗 )

6. PV-DM (Distributed Memory for Paragraph Vector)
Given 𝑝 𝑖 = 𝑤 1 𝑤 2 … 𝑤 𝑛 , find their vector embedding 𝑞 𝑖 , 𝑣 1 ,…, 𝑣 𝑛 , 𝑣 1 , …, 𝑣 𝑛 such that 𝑃 𝑤 4 𝑝 𝑖 𝑤 1 𝑤 2 𝑤 3 =softmax 𝑣 4 𝑇 𝑞 𝑖 𝑣 1 𝑣 2 𝑣 3
A bias term 𝑏 4 may be added
Train 𝑞 𝑖 , 𝑣 𝑗 , 𝑣 𝑗 vectors on large text corpus to maximize the log likelihood 𝑖,𝑗 ∈𝐷 log 𝑃( 𝑤 𝑖 | 𝑝 𝑗 )
( 𝑣 4 , 𝑏 4 )
Concatenate/average
𝑞 𝑖
𝑣 1
𝑣 2
𝑣 3

7. Results of [Le & Mikolov 2014]
PV-DM works better than PV-DBOW
Slight improvement when used together
12.2% error rate for Stanford sentiment analysis task using PV
20% improvement from state-of-the-art
7.42% error rate for IMDB review sentiment analysis task
15% improvement from state-of-the-art
3.82% error rate for paragraph similarity task
32% improvement from state-of-the-art
Vector embedding works well at the sentence/paragraph level!

8. Vector Embedding
Today, word embedding is used as the first step in almost all NLP tasks
Word2Vec, GloVe, ELMO, BERT, …
In general, “vector embedding” is an extremely hot research topic for many different types of datasets
Graph embedding
User embedding
Time-series data embedding …

9. Any Questions?
Next topic: language models

10. Language Model: A Brief Recap
Given a sequence of words 𝑤 1 𝑤 2 … 𝑤 𝑛 , assign probability 𝑃( 𝑤 1 𝑤 2 … 𝑤 𝑛 )
Q: How can we compute 𝑃( 𝑤 1 𝑤 2 … 𝑤 𝑛 )?
A: For each word sequence 𝑤 1 𝑤 2 … 𝑤 𝑛 , count their frequency in the corpus, #( 𝑤 1 𝑤 2 … 𝑤 𝑛 ) and divide it by the corpus size 𝑁
Divide by (𝑁−𝑛+1) to be precise
Challenge: Data Sparsity
For any reasonably large 𝑛, #( 𝑤 1 𝑤 2 … 𝑤 𝑛 ) is likely to be zero

11. Chain Rule and N-gram Language Model
𝑃 𝑤 1 𝑤 2 … 𝑤 𝑛 =𝑃 𝑤 1 𝑃 𝑤 2 𝑤 1 𝑃 𝑤 3 𝑤 1 𝑤 2 …𝑃 𝑤 𝑛 𝑤 1 … 𝑤 𝑛−1
For 3-gram model, approximate 𝑃 𝑤 𝑖 𝑤 1 … 𝑤 𝑖−1 ≈𝑃 𝑤 𝑖 𝑤 𝑖−2 𝑤 𝑖−1
Then, 𝑃 𝑤 1 𝑤 2 … 𝑤 𝑛 ≈𝑃 𝑤 1 𝑃 𝑤 2 𝑤 1 𝑃 𝑤 3 𝑤 1 𝑤 2 …𝑃 𝑤 𝑛 𝑤 𝑛−2 𝑤 𝑛−1
Q: Good first-order approximation. How can we make it better?
Q: When does it work well? When does it fail? How can we avoid failure cases?

12. Ideas for n-Gram Model Improvement
Use longer n-grams!
Higher-order n-gram
Estimates from low frequency n-grams are inherently inaccurate
“Back-off” to lower-order estimate if zero count
Smoothing: Laplace, Jelinek-Mercer, absolute discount, Kats, Kneser-Ney
Skip missing word in sequence: skipping
Use context-specific language model
People tend to use words that they used before: caching
Build a separate language model for each sentence/document type: sentence- mixture model
Use word classes
Clustering: If “on Tuesday” is frequent, “on Wednesday” is also likely to be frequent

13. Dealing with Zero-Frequency Grams
Q: Some 3-grams, say 𝑎𝑏𝑐, do not appear in our corpus. Should we set 𝑃 𝑐 𝑎𝑏 =0?
“Back-off policy”
Use the estimate from a lower-order n-gram
𝑃 𝑐 𝑎𝑏 = 𝑃 𝑐 𝑎𝑏 if # 𝑎𝑏𝑐 >0 𝑃 𝑐 𝑏 if # 𝑏𝑐 >0 𝑃 𝑐 if # 𝑐 > /𝑁 otherwise.

14. Improving Low-Frequency Estimates
Estimates from low frequency gram is inherently inaccurate
Zero-count estimate is an extreme example of this inaccuracy
A word/gram cannot appear 1.3 times!
Smoothing: class of techniques that try to improve low-frequency estimates
“Smooth out” low frequency estimates, while “preserving” how frequency estimates
Q: How?

15. Laplace Smoothing Add 1 to all gram frequency counts
𝑃 𝐿𝑎𝑝𝑙𝑎𝑐𝑒 𝑐 𝑎𝑏 = #(𝑎𝑏𝑐)+1 𝑁+𝑉
#(𝑎𝑏𝑐): gram frequency
𝑁: corpus size
𝑉: vocabulary size
Assume every gram has been seen once even before we see any
Q: Why did people think it was a good idea? A:
Assign non-zero probability to zero-frequency grams
High-frequency gram’s estimate is close to the original estimate

16. Jelinek-Mercer Smoothing (a.k.a. Simple Interpolation)
Back off uses low-order estimate only for zero frequency. Let us use it more!
“Mix” higher-order n-gram estimate with low-order n-gram estimates
𝑃 𝐽𝑀 (𝑐|𝑎𝑏) = 𝜆 1 𝑃(𝑐|𝑎𝑏)+ 𝜆 2 (𝑐|𝑏)+ 𝜆 3 𝑃 𝑐 + 𝜆 4 /𝑉 = 𝜆 1 #(𝑎𝑏𝑐) #(𝑎𝑏) 𝜆 2 #(𝑏𝑐) #(𝑏) 𝜆 3 #(𝑐) 𝑁 + 𝜆 4 1 𝑉 with 𝜆 1 + 𝜆 2 + 𝜆 3 + 𝜆 4 =1
“Smoothed version” of back-off

17. Absolute Discount Smoothing
“Subtract” a constant 𝑑 from each frequency and “mix-in” lower-order estimate
𝑃 𝐴𝐷 (𝑐|𝑎𝑏) = max⁡(#(𝑎𝑏𝑐)−𝐷,0) #(𝑏𝑐) +𝛼 𝑃 𝐴𝐷 (𝑎|𝑏) 𝛼 is chosen so that 𝑤 𝑃 𝐴𝐷 (𝑤|𝑎𝑏) =1
Q: Why absolute discount? Why did people think thought it would be a good idea? A:
When #(𝑎𝑏𝑐) is high, 𝑃 𝐴𝐷 is close to non-smoothed estimate
When #(𝑎𝑏𝑐) is low, the lower-order estimate becomes important

18. Kneser-Ney Smoothing Problem of absolute discounting
Suppose the phrase San Francisco shows up many times in training corpus
We might have seen a lot more San Francisco than glasses
#(Francisco) > #(glasses)
Let’s predict: I can’t see without my reading ___.
𝑃 𝐴𝐷 Francisco reading = max⁡(#(reading Francisco)−𝑑,0) #(reading) +𝛼 max⁡(#(Francisco)−𝑑,0)
Very likely, #(reading Francisco) < d, and so the second term, #(Francisco), will dominate.
We might end up predicting the blank to be Francisco!
Problem: Francisco appears mostly behind San, but not others!
Glasses may appear behind many different words
Q: How can we incorporate this information?

19. Kneser-Ney Smoothing
Assign higher probability if a word appears behind many different words
Make the probability proportional to the # of previous context words
𝑃 𝐾𝑁 𝑤 𝑖 𝑤 𝑖−2 𝑤 𝑖−1 = max⁡(#( 𝑤 𝑖−2 𝑤 𝑖−1 𝑤 𝑖 )−𝑑,0) #( 𝑤 𝑖−2 𝑤 𝑖−1 ) +𝛼 𝑣 #(𝑣 𝑤 𝑖 )>0 𝑤 𝑣 #(𝑣𝑤)>0
Modified Kneser-Ney smoothing
Use different discount 𝑑 per each gram count

20. Katz Smoothing
The estimates for grams that appear only once in the corpus are likely to be higher than their true values
A gram cannot appear only 0.2 times!
We need to “discount” their estimates appropriately
Q: By how much?
“Good-Turing Frequency Estimator”
Any n-gram that occurs 𝑟 times should be discounted by 𝑑𝑖𝑠𝑐 𝑟 = 𝑟+1 𝑛 𝑟+1 𝑛 𝑟 ( 𝑛 𝑟 :# 𝑛grams that occur 𝑟 times)
Katz Smoothing: intuition
𝑃 𝐾𝑎𝑡𝑧 𝑐 𝑎𝑏 = 𝑑𝑖𝑠𝑐(#𝑎𝑏𝑐) #𝑎𝑏

21. Katz Smoothing
𝑃 𝐾𝑎𝑡𝑧 𝑐 𝑎𝑏 = 𝑑𝑖𝑠𝑐(#(𝑎𝑏𝑐)) #(𝑎𝑏) if # 𝑎𝑏𝑐 >0 𝛼 𝑃 𝐾𝑎𝑡𝑧 𝑐 𝑏 otherwise

22. Skipping
Basic idea: Why do we have to drop the farthest word if frequency is low? What about “told Jelinek news”?
“Jelinek news” is still infrequent because “Jelinek” is infrequent word!
“Skip” or “ignore” middle words
𝑃 𝑠𝑘𝑖𝑝 𝑑 𝑎𝑏𝑐 = 𝜆 1 𝑃 𝑑 𝑎𝑏𝑐 + 𝜆 2 𝑃 𝑑 𝑏𝑐 + 𝜆 3 𝑃 𝑑|𝑎𝑐 + 𝜆 4 𝑃 𝑑|𝑎𝑏 with 𝜆 1 + 𝜆 2 + 𝜆 3 + 𝜆 4 =1

23. Caching: User-specific Language Model
A person tend to use the same words that she used before
Build a “user-specific” language model and mix it with the general language model
𝑃 𝑐𝑎𝑐ℎ𝑒 𝑤 𝑖 𝑤 𝑖−2 𝑤 𝑖−1 = 𝜆 1 𝑃 𝑔𝑒𝑛𝑒𝑟𝑎𝑙 𝑤 𝑖 𝑤 𝑖−2 𝑤 𝑖−1 + 𝜆 2 𝑃 𝑢𝑠𝑒𝑟 𝑤 𝑖 𝑤 𝑖−2 𝑤 𝑖−1
“User” can be “recent past”
“User model” from past 1000 words

24. Sentence-Mixture Model
Health news language model should be very different from Twitter language model
Build separate language model for each type of corpus
Use appropriate model if the source is known
Mix the models if the source is unknown
𝑃 𝑚𝑖𝑥 𝑤 𝑖 𝑤 𝑖−2 𝑤 𝑖−1 = 𝜆 1 𝑃 Health 𝑤 𝑖 𝑤 𝑖−2 𝑤 𝑖−1 + 𝜆 2 𝑃 Twitter 𝑤 𝑖 𝑤 𝑖−2 𝑤 𝑖−1
“Different source” can be different sentence types
Questions, exclamation, statements,…

25. Summary: Language Model
Caching provides the most significant accuracy improvement
But collecting accurate user-specific statistics is often difficult and the complexity of the model increases significantly
Among smoothing techniques, modified Kneser-Ney performs best in practice
Sentence-mixture model also works well in practice
But requires significantly more training data and increases the model complexity
Combining all three techniques do work

26. Announcement Start preparing your presentation
Your presentations will start next week!
Decide project ideas for your group
Project proposal is due by 5th week
Project idea presentation during the 5th week
Working on a Kaggle competition can be an option
If you get stuck,
I am available to help
Leverage my office hour to your advantage