Neural Language Model CS246 Junghoo “John” Cho

Statistical Language Model The probability of word sequence 𝑤 1 ,…, 𝑤 𝑛 : 𝑃 𝑤 1 ,…, 𝑤 𝑛 Q: Why is it useful? A: Many applications 𝑃 𝑤 𝑤 1 ,…, 𝑤 𝑛 = 𝑃( 𝑤 1 ,…, 𝑤 𝑛 , 𝑤) 𝑃( 𝑤 1 ,…, 𝑤 𝑛 ) word prediction, spell correction, … Q: How can we “learn” the language model 𝑃 𝑤 1 ,…, 𝑤 𝑛 ?

Learning Language Model Input corpus: 1 billion word sequence. 10,000 word vocabulary. Q: How many times each word 𝑤 𝑖 (unigram) be seen on average? Will the estimated 𝑃 𝑤 𝑖 be reliable? Q: What about 𝑤 𝑖 𝑤 𝑗 (bigram)? Q: What about 𝑤 𝑖 𝑤 𝑗 𝑤 𝑘 (trigram)? The quality of learned language model depends heavily on the “quality” and “quantity” of the input corpus

Generalization Q: How can we “estimate” 𝑃 𝑤 1 ,…, 𝑤 𝑛 for longer 𝑤 1 ,…, 𝑤 𝑛 ? A: Chain rule and approximation by n-gram 𝑃 𝑤 1 ,…, 𝑤 𝑛 =𝑃 𝑤 1 𝑃 𝑤 2 𝑤 1 𝑃 𝑤 3 𝑤 1 , 𝑤 2 …𝑃( 𝑤 𝑛 | 𝑤 1 ,…, 𝑤 𝑛−1 ) Unigram model: 𝑃 𝑤 𝑖 | 𝑤 𝑗 ,… 𝑤 𝑘 ≅𝑃 𝑤 𝑖 bigram model: 𝑃 𝑤 𝑖 | 𝑤 𝑗 ,… 𝑤 𝑘 ≅𝑃 𝑤 𝑖 |𝑤 𝑗 N-gram model: 𝑃 𝑤 𝑖 | 𝑤 𝑗 ,… 𝑤 𝑘 ≅𝑃 𝑤 𝑖 |𝑤 𝑗 ,…, 𝑤 𝑗+𝑛−1 In practice, N is at most 3 due to high dimension of 𝑤 𝑖 ′ s This was the state-of-the-art until early 2000’s [Goodman 2001] Q: Can we estimate 𝑃 𝑤 1 ,…, 𝑤 𝑛 for long 𝑤 1 ,…, 𝑤 𝑛 better?

Neural Probabilistic Language Model [Bengio2003] Learn the probability function 𝑃 𝑤| 𝑤 1 ,… 𝑤 𝑛 using neural network! Represent each word 𝑤 𝑖 to an m-dimensional vector 𝑣 𝑖 Express 𝑃 𝑤| 𝑤 1 ,… 𝑤 𝑛 as a function of 𝑣 𝑖 ′ 𝑠: 𝑓(𝑤, 𝑣 1 ,…, 𝑣 𝑛 ) 𝑖 𝑓( 𝑤 𝑖 , 𝑣 1 , 𝑣 𝑛 )=1 Learn the function 𝑓(𝑤, 𝑣 1 ,…, 𝑣 𝑛 ) and vector representation 𝑣 𝑖 ’s of word 𝑤 𝑖 ’s simultaneously by learning on a neural network

Example 4 words, 2d vector representation 𝑤 1 = 0.1, 0.8 𝑤 2 = 0.3, 0.2 𝑤 3 = 0.7, 0.1 𝑤 4 = 0.6, 0.4 𝑃 𝑤 𝑖 𝑤 3 𝑤 2 𝑤 4 =𝑓 𝑤 𝑖 , 0.7, 0.1 , 0.3, 0.2 , 0.6, 0.4 𝑓 𝑤 1 , 0.7, 0.1 , 0.3, 0.2 , 0.6, 0.4 =0.6 𝑓 𝑤 2 , 0.7, 0.1 , 0.3, 0.2 , 0.6, 0.4 =0.2 𝑓 𝑤 3 , 0.7, 0.1 , 0.3, 0.2 , 0.6, 0.4 =0.1 𝑓 𝑤 4 , 0.7, 0.1 , 0.3, 0.2 , 0.6, 0.4 =0.1 𝑓() is a “vector function” that maps a sequence of input word vectors into an output probability vector of V-dimension

Probability Function 𝑓 ( 𝑣 1 ,…, 𝑣 𝑛 ) Function from 𝑛 m-dimensional vectors to a V-dimensional probability vector Each dimension in the output vector represents the probability of each word 𝑤 𝑖 Q: How can we represent 𝑓 ( 𝑣 1 ,…, 𝑣 𝑛 )? 𝑓 𝑣 1 𝑣 2 𝑣 𝑛 ⋮ 𝑤 1 :0.03 𝑤 2 :0.01 ⋮ 𝑤 𝑉 :0.05

Word to Vector Mapping V words to N-dimensional vectors Can be represented as mxV matrix, where each column corresponds to the vector representation of each word Can be interpreted as mapping from 1-hot-encoding of a word to its m- dimensional vector encoding | | 𝑣 1 𝑣 2 … | | | 𝑣 𝑉 | 0 ⋮ 1 ⋮ 0 = | 𝑣 𝑖 |

Neural Network of 𝑓 ( 𝑣 1 ,…, 𝑣 𝑛 ) in [Bengio 2003] 𝑤 1 … … 𝑣 1 𝑣 𝑛 [m] W … [h] ℎ 𝑣 𝑖 =𝑊 𝑤 𝑖 ℎ =𝐻 𝑣 1 ⋮ 𝑣 𝑛 𝑦 =𝑈 tanh ℎ 𝑓 =softmax( 𝑦 ) H . 𝑦 … [V] softmax U . [V] tanh 𝑤 𝑛 … tanh 𝑥 = 𝑒 𝑥 − 𝑒 −𝑥 𝑒 𝑥 + 𝑒 −𝑥 softmax 𝑥 𝑖 = 𝑒 𝑥 𝑖 𝑖 𝑒 𝑥 𝑖 [V]

Learning 𝑓 (and 𝑣 )[Bengio 2003] Use back-propagation algorithm to maximize the log-likelihood of the training data For every training instance, the entire matrix of U (all h x V entries), H (all n x m x h entries), and W (just n x m entries) are updated Q: V = 20,000, n = 10, m=50, h = 100. How many parameter updates in each layer? Updating U is computationally most expensive Parallelized U update algorithm for fast learning

Result of [Bengio 2003] Obtained state-of-the-art result on 15 million word input data in terms of perplexity 10-20% better than smoothed trigram model A lot of excitement due to another significant improvement due to “neural network” Much follow-up work was done after this paper