Deep Learning in NLP Word representation and how to use it for Parsing For Sig Group Seminar Talk Wenyi Huang harrywy@gmail.com

Existing NLP Applications Language Modeling Speech Recognition Machine Translation Part-Of-Speech Tagging Chunking Named Entity Recognition Semantic Role Labeling Sentiment Analysis Paraphrasing Question-Answering Word-Sense Disambiguation

Word Representation One hit representation: movie [0 0 0 0 0 0 0 0 1 0 0 0 0] && film [0 0 0 0 0 1 0 0 0 0 0 0 0] = 0 Distributional similarity based representations You can get a lot of value by representing a word by means of its neighbors: Class-based (hard) clustering word representations Brown clustering (Brown et al. 1992) Exchange clustering (Martin et al. 1998, Clark 2003) Soft clustering word representations LSA/LSI LDA

Brown Clustering: Example Clusters (from Brown et al, 1992)

The Formulation Quality of a partition 𝐶: 𝑉 is the set of all words seen in the corpus 𝑤 1 , 𝑤 2 … 𝑤 𝑇 Say 𝐶 : 𝑉 → 1, 2, …𝑘 is a partition of the vocabulary into 𝑘 classes The model: 𝑝 𝑤 1 , 𝑤 2 ,… 𝑤 𝑡 = 𝑖=1 𝑛 𝑝 𝑤 𝑖 𝐶 𝑤 𝑖 𝑝 𝐶 𝑤 𝑖 𝐶 𝑤 𝑖−1 ) Quality of a partition 𝐶: 𝑄 𝐶 = 1 𝑛 𝑖=1 𝑛 log 𝑝 𝑤 𝑖 𝐶 𝑤 𝑖 𝑝(𝐶( 𝑤 𝑖 )|𝐶( 𝑤 𝑖−1 )) Bigram model, HHM

A Sample Hierarchy (from Miller et al., NAACL 2004) Huffman Coding Prefixes -> similarity

Language Model N-gram model (n=3) 𝑃 𝐼, 𝑠𝑎𝑤, 𝑡ℎ𝑒, 𝑟𝑒𝑑, ℎ𝑜𝑢𝑠𝑒 ≈𝑃 𝐼 <𝑠>,<𝑠> 𝑃 𝑠𝑎𝑤 <𝑠>,𝐼 𝑃 𝑡ℎ𝑒 𝐼,𝑠𝑎𝑤 𝑃 𝑟𝑒𝑑 𝑠𝑎𝑤,𝑡ℎ𝑒 𝑃 ℎ𝑜𝑢𝑠𝑒 𝑡ℎ𝑒,𝑟𝑒𝑑 𝑃(</𝑠>|𝑟𝑒𝑑,ℎ𝑜𝑢𝑠𝑒) Calculated from n-gram frequency counts: 𝑃 𝑤 𝑖 𝑤 𝑖− 𝑛−1 ,…, 𝑤 𝑖−1 = 𝑐𝑜𝑢𝑛𝑡 𝑤 𝑖− 𝑛−1 ,…, 𝑤 𝑖−1 , 𝑤 𝑖 𝑐𝑜𝑢𝑛𝑡 𝑤 𝑖− 𝑛−1 ,…, 𝑤 𝑖−1

A Neural Probabilistic Language Model (Bengio et al, NIPS’2000 and JMLR 2003) Motivation: LM does not take into account contexts farther than 2 words. LM does not take into account the “similarity” between words. Idea: A word 𝑤 is associated with a distributed feature vector (a real-valued vector in ℝ 𝑛 𝑛 is much smaller than size of the vocabulary) Express joint probability function 𝑓 of words sequence in terms of feature vectors Learn simultaneously the word feature vector and the parameters of 𝑓

Neural Language Model Neural architecture 𝑓 𝑖, 𝑤 𝑡−1 ,…, 𝑤 𝑡−𝑛+1 =𝑔(𝑖,𝐶 𝑤 𝑡−1 ,…,𝐶( 𝑤 𝑡−𝑛+1 ))

Neural Language Model softmax output layer: 𝑃 𝑤 𝑡 𝑤 𝑡−1 ,⋯, 𝑤 𝑡−𝑛+1 = 𝑒 𝑦 𝑤 𝑡 𝑖 𝑒 𝑦 𝑖 𝑦 𝑖 unnormalized log-probabilities for each output word 𝑖 𝑦=𝑏+𝑈tanh(𝑑+𝐻𝑥) 𝑥 is the word features layer activation vector 𝑥= 𝐶 𝑤 𝑡−1 ,…,𝐶 𝑤 𝑡−𝑛+1 The free parameters of the model are: 𝜃=(𝑏,𝑑,𝑊,𝑈,𝐻,𝐶) 𝑏 output biases (|𝑉|) 𝑑 the hidden layer biases (ℎ) 𝑈 the hidden-to-output weights ( 𝑉 ×ℎ) 𝐻 the hidden layer weights (ℎ×(𝑛−1)𝑚) 𝐶 word features ( 𝑉 ×𝑚) 4 weeks of training (40 CPUs) on 14,000,000 words training set |V|=17964

Neural word Embedding as a distributed representation http://metaoptimize.com/projects/wordreprs/

A neural network for learning word vectors (Collobert et al. JMLR 2011) A word and its context is a positive training sample; a random word in that same context gives a negative training sample: [+] positive = Score(Cat chills [on] a mat) > [-] negative = Score(Cat chills [god] a mat) What to feed in the NN each word is an n-dimensional vector, a look up table: 𝐿∈ ℝ 𝑛× 𝑉 Training objective: 𝜃→ 𝑀𝑎𝑥{0, 1 − 𝑆 𝑝𝑜𝑠 + 𝑆 𝑛𝑒𝑔 } 3-layer NN: 𝑠= 𝑈 𝑇 𝑓 𝜃 (𝑊𝑥+𝑏) Where 𝑓 𝜃 ⋅ is a NN function. SENNA: http://ml.nec-labs.com/senna/ Window size n = 11 |V| = 1300000 7 weeks

Linguistic Regularities in Continuous Space Word Representations (Mikolov, et al. 2013) Recurrent Neural Network Model d(t) 0 0 1 …. 0 𝑣 Error= y(t)-d(t) 0 1 0 …. 0 𝑣 One-hit representation Context at time t-1

Linguistic Regularities in Continuous Space Word Representations (Mikolov, et al. 2013) Recurrent Neural Network Model The input vector 𝑤(𝑡) represents input word at time 𝑡 encoded using One hit coding. The output layer 𝑦(𝑡) produces a probability distribution over words. The hidden layer 𝑠(𝑡) maintains a representation of the sentence history. 𝑤(𝑡) and 𝑦 𝑡 are of same dimension as vocabulary Model: 𝑠 𝑡 =𝑓 𝑈𝑤 𝑡 +𝑊𝑠 𝑡−1 𝑦 𝑡 =𝑔(𝑉𝑠(𝑡)) Where 𝑓 is the sigmod function and 𝑔 is the softmax funciton

Linguistic Regularities in Continuous Space Word Representations (Mikolov, et al. 2013) Training: Stochastic Gradient Descent (SGD) Objective(Error) function: 𝑒𝑟𝑟𝑜𝑟 𝑡 =𝑑 𝑡 −𝑦(𝑡) where d(t) is the desired vector, i.e w(t) Go through all the training data iteratively, and update the weight matrices U, V and W online (after processing every word) Training is performed in several epochs (usually 5-10) Where is the word representation? 𝑈, with each column

Linguistic Regularities in Continuous Space Word Representations (Mikolov, et al. 2013) Measuring Linguistic Regularity Syntactic/Semetic Test These representations are surprisingly good at capturing syntactic and semantic regularities in language, and that each relationship is characterized by a relation-specific vector offset.

Exploiting Similarities among Languages for Machine Translation (Mikolov, et al. 2013 http://arxiv.org/pdf/1309.4168.pdf) Figure 1: Distributed word vector representations of numbers and animals in English (left) and Spanish (right). The five vectors in each language were projected down to two dimensions using PCA, and then manually rotated to accentuate their similarity. It can be seen that these concepts have similar geometric arrangements in both spaces, suggesting that it is possible to learn an accurate linear mapping from one space to another.

Improving Word Representations via Global Context and Multiple Word Prototypes (Huang, et al. ACL 2013)

Improving Word Representations via Global Context and Multiple Word Prototypes (Huang, et al. ACL 2013) Improve Collobert & Weston’s model Training objective: 𝜃→ 𝑀𝑎𝑥{0, 1 − 𝑆 𝑝𝑜𝑠 + 𝑆 𝑛𝑒𝑔 } ↓ 𝜃→ 𝑀𝑎𝑥{0, 1 − 𝑆 𝑝𝑜𝑠, 𝑑 + 𝑆 𝑛𝑒𝑔,𝑑 } where 𝑑 is the document (weighted sum of words in 𝑑)

Improving Word Representations via Global Context and Multiple Word Prototypes (Huang, et al. ACL 2013) The Model Clustering word base on context and retrain the model

Summary: Word Representation and Language Model Bengio et al, 2010 Associated Press (AP) News from 1995 and 1996 14,000,000 words Word: 17964 Trained for 4 weeks (40 CPUS) C&W English Wikipedia  + Reuters RCV1 Words: 130000 Dimensions: 50, Trained for 7 weeks Mikolov Broadcast news Words: 82390  Dimensions: 80, 640, 1600 Trained for several days Huang 2012 English Wikipedia Words: 100232 Dimensions: 50 |V|= 6000 10 cluster for each words

Parsing What we want:

Using word vector space model The meaning (vector) of a sentence is determined by (1) The meanings of its words and (2) The rules that combine them.

Recursive Neural Networks for Structure Prediction Inputs: two candidate children’s representations Outputs: The semantic representation if the two nodes are merged Score of how plausible the new node would be

Recursive Neural Networks

Parsing Example with an RNN

Parsing Example with an RNN

Parsing Example with an RNN

Parsing Example with an RNN

Parsing with Compositional Vector Grammars (Socher, et al. ACL 2013) A Compositional Vector Grammar (CVG) model, which combines PCFGs with a syntactically untied recursive neural network that learns syntactico-semantic, compositional vector representations.

Parsing with Compositional Vector Grammars (Socher, et al. ACL 2013) Probabilistic Context-Free Grammars (PCFGs) A PCFG consists of: A context-free grammar G = (N, Σ, S, R). A parameter 𝑞 𝛼 → 𝛽 for each rule 𝛼 → 𝛽 ∈ 𝑅. The parameter 𝑞(𝛼 → 𝛽) can be interpreted as the conditional probabilty of choosing rule 𝛼 → 𝛽 in a left-most derivation, given that the non-terminal being expanded is 𝛼. For any 𝑋 ∈ 𝑁, we have the constraint: 𝑎→𝛽∈𝑅, 𝛼=𝑋 𝑞 𝛼→𝛽 =1 Given a parse-tree 𝑡 ∈ 𝑇 𝐺 containing rules 𝛼 1 → 𝛽 1 , 𝛼 2 → 𝛽 2 , . . . , 𝛼 𝑛 → 𝛽 𝑛 , the probability of 𝑡 under the PCFG is 𝑝 𝑡 = 𝑖=1 𝑛 𝑞 𝛼→𝛽 Chomsky Normal Form -> Binary Parsing Tree

Parsing with Compositional Vector Grammars (Socher, et al. ACL 2013) Define a structured margin loss Δ 𝑦 𝑖 , 𝑦 for predicting a tree 𝑦 for a given correct tree. Δ 𝑦 𝑖 , 𝑦 = 𝑑∈𝑁( 𝑦 ) 𝑘×1{𝑑∉𝑁( 𝑦 𝑖 )} For a given set of training instances ( 𝑥 𝑖 , 𝑦 𝑖 ), we search for the function 𝑔 𝜃 , parameterized by 𝜃, with the smallest expected loss on a new sentence. 𝑔 𝜃 = argmax 𝑦 ∈𝑌(𝑥) 𝑠(𝐶𝑉𝐺 𝜃,𝑥, 𝑦 ) where 𝑠 𝐶𝑉𝐺 𝜃, 𝑥 𝑖 , 𝑦 𝑖 ≥𝑠 𝜃, 𝑥 𝑖 , 𝑦 +Δ 𝑦 𝑖 , 𝑦

Parsing with Compositional Vector Grammars (Socher, et al. ACL 2013) The CVG computes the first parent vector via the SU-RNN: 𝑃 (1) =𝑓 𝑊 (𝐵,𝐶) 𝑏 𝑐 where 𝑊 (𝐵,𝐶) ∈ ℝ 𝑛×2𝑛 is now a matrix that depends on the categories of the two children. Score for each node consists of summing two elements: 𝑠 𝑝 1 = 𝑣 𝐵,𝐶 𝑇 𝑝 1 + log 𝑃( 𝑃 1 →𝐵 𝐶) where 𝑣∈ ℝ 𝑛 is a vector of parameters that need to be trained. And 𝑃( 𝑃 1 →𝐵 𝐶) comes from the PCFG

Parsing with CVGs bottom-up beam search keeping a k-best list at every cell of the chart The CVG improves the PCFG of the Stanford Parser by 3.8% to obtain an F1 score of 90.4%. As a reranker.

Q&A Thanks!