Sentence Modeling Representation of sentences is the heart of Natural Language Processing A sentence model is a representation and analysis of semantic content of a sentence for classification or generation The sentence modeling task is at the core of many tasks such as sentiment analysis, paraphrase detection, entailment recognition, summarization, discourse analysis, machine translation, grounded language learning and image retrieval The aim of sentence modeling is a feature function that guides the process by which features of a sentence are extracted.

One Dimensional Convolution

M-gram Dot product Vector of weights Size: m Filter Input Sequence Sentence

Produces a sequence c

Narrow Convolution Size of c : s – m + 1 It requires that s ≥ m

Wide Convolution Size: s + m - 1

Wide Convolution Size of c s+m-1 No requirement on s or m Out of range values are taken to be 0 Result of narrow convolution is subsequence of result of wide convolution

Advantages of Wide Convolution Guarantees that a valid non empty c will always be produced All weights in the filter reach the entire sentence Holds no limit on the size of m or s

Time Delay Neural Network A key feature for TDNN’s are the ability to express a relation between inputs in time. The sequence s is viewed as having a time dimension and the convolution is applied over the time dimension.

Max TDNN

Properties of Max TDNN Sensitive to order of the words Does not depend on external language specific features Largely uniform importance to the signal from each of the words Range of feature detectors is limited Higher order and long range feature detectors cannot be incorporated Multiple occurrences of features and the sequence ignored Pooling factor: s-m+1

k-Max Pooling

k – Max Pooling Given a value k and a sequence P of length p, k-max pooling selects the subsequence p-max of the k highest values of p. The order of the values in p-max corresponds to the original order in p.

k-Max Pooling k most active features Features may be number of positions apart Preserves the order of the features But is insensitive to their specific positions Can detect multiple occurrences of feature

What should k be? Why not let it decide for itself?

Dynamic k-Max Pooling Suppose length of sentence = 18 L = 3 Ktop = 3

Second Order Feature Map Multiple Feature Maps Feature Map Second Order Feature Map Convolution K-max Pooling Layer Non linear function

To increase the number of learnt feature detectors of a certain order, multiple feature maps may be computed in parallel at the same layer.

Folding Feature detectors in different rows are independent of each other.

Properties of Sentence Model The subsequence of n-grams extracted by the pooling operation induces invariance to absolute positions, but maintains their order and relative positions. DCNN feature graphs have a global range of the pooling operations DCNN has internal input dependent structure and does not rely on externally provided parse trees.

Experiments

Sentiment Prediction in Movie Reviews Concerns prediction of sentiment of movie reviews in Stanford Sentiment Treebank Output is binary in experiment 1 and “negative, somewhat negative, neutral, somewhat positive, positive” in experiment 2 Binary: MultiClass:

Question Type Classification TREC question dataset Six Different Question Types

Twitter Sentiment Prediction with Distant Supervision Large dataset of tweets Tweet is labelled positive or negative automatically based on emoticon Tweets are preprocessed

Conclusion Dynamic CNN defined, which uses Dynamic k-max Pooling Feature Graph captures word relation of varying size High performance on sentiment prediction and question classification without requiring external features