From Frequency to Meaning: Vector Space Models of Semantics Peter D. Turney, Patrick Pantel

VSM Applications Originally developed for SMART information retrieval system Used VSMs to find documents related to a search VSM techniques used in modern search engines More recently being applied to multiple choice question answering (TOEFL, SAT)

Why VSMs? Extract knowledge automatically from corpora, so don't require hand coded knowledge or ontologies Word similarity tasks typically require large lexicons (like WordNet), which are much harder to come by the large corpora VSMs are successful in measuring similarity of meaning between words, phrases, and documents

Types of VSMs Types of matrices: Term-document Word-context Pair-pattern

Term-Document VSM Row Vectors : Terms Column Vectors : Documents

Term-Document VSM Relies upon the bag of words hypothesis: -The frequency of words in a document tend to indicate the relevance of the document to a query. The column vector in a term-document matrix tends to indicate what the document is about.

Word-Context VSM Rows : Words Columns : Context Essentially the same as term-documents, but focused on the row vectors

Word-Context VSM Relies upon the distributional hypothesis. -Words that occur in similar contexts tend to have similar meanings. Row vectors in a word-context matrix indicate similar word meanings.

Pair-Pattern VSM Rows : pairs of words (mason:stone, carpenter:wood) Columns : patterns (“X cuts Y”, “X works with Y”) X cuts Y | X works with Y | X breaks Y mason:stone 3 2 3 carpenter:wood 2 2 3 mason:wood 0 0 0

Pair-Pattern VSM Similarity of column in a pair-pattern matrix indicates similarity of the patterns. Extended distributional hypothesis : Patterns that co-occur with similar pairs tend to have similar meanings. The pattern “X solves Y” and “Y is solved by X” tend to co-occur with the same pairs, which indicates that they have similar meanings.

Pair-Pattern VSM Similarity of rows in a pair-pattern matrix indicate similarity of word pairs. Latent relation hypothesis: Pairs of words that co-occur in similar patterns tend to have similar semantic relations. Inverse of previous hypothesis.

Linguistic Processing Tokenization Normalization Annotation

Mathematical Processing Generate frequencies from corpora Adjust weights of elements in matrix Smoothing Measure similarity of vectors

Building Frequency Matrix Scan through corpus, recording events and frequencies into a hash table/database. Use resulting data structure to build a frequency matrix.

Weighting Elements Give more weight to surprising events and less weight to expected events Surprising events are more discriminative of semantic similarity In the context of 'rat' and 'mouse', 'dissect' and 'exterminate' are much more discriminative than 'have' and 'like'

Weighting Elements Term frequency * inverse document frequency Element gets a high weight if the corresponding term occurs frequently in the corresponding document, but rare in other documents. Pointwise Mutual Information (PMI) Length normalization Long documents tend to be favored, which is correct by length normalization Term weighting Terms like 'hostage' and 'hostages' tend to be correlated but not normalized to the same term, so their weights are reduced when the co-occur in a document.

Smoothing Improve performance by limiting the number of vector components Vectors that do not share non-zero coordinates are unrelated, and can be ignored. Smoothing algorithms remove elements below certain weights to zero and keeps elements that are more relevant.

Singular Value Decomposition Decomposes the matrix into three matrices (U,Ʃ,V), where two are in column orthonormal form and the third is a diagonal matrix of singular values. Ʃ : Diagonal matrix of top k values. U, V : Corresponding columns

Comparing the Vectors Most popular way : take the cosine Vectors of frequent words tend to be long, while rare word vectors tend to be short. Cosine ensures vector length is irrelevant to their similarity, as it measures only the angle between them.

Efficient Comparisons Sparse Matrix Multiplication -Throw out vectors that don't share non-zero coordinates Distributed Implementation using MapReduce Randomized Algorithms -Scale large vectors to small vectors while losing minimal information

Implementations Term-Document Matrix : Lucene Content – Webpages, PDF documents, images, video, etc Fields – Developer defined parts of these Content elements. Columns : Documents Fields : Rows

Implementations Word-Context Matrix Semantic Vectors Open-source project to implement VSMs and random projection to measure word similarity. Uses Lucene to create a term-document matrix, and uses random projection to reduce dimensions.

Implementations Pair-Pattern Matrix Latent Relational Analysis Builds pair-pattern matrix using a textual corpus as input. Uses WordNet to mitigate spareness: <Korea, Japan> expanded to <South Korea, Japan>, <Republic of Korea, Japan>, <Korea, Nippon>, etc

Applications Term-Document: Document Retrieval Document Clustering Document Classification Essay grading Document segmentation QA Call Routing

Applications Word-Context: Word similarity Word clustering Word classification Automatic thesaurus generation WSD Context-sensitive spelling correction SRL Query expansion Textual advertising Information Extraction/NER

Applications Pair-pattern: Relational similarity Pattern similarity Relational clustering Relational classification Relational search Automatic thesaurus generation Analogical mapping

The Future VSMs typically don't account for word order (pair- pattern matrices do). Some people (Clark and Pulman 2007, Widdows and Ferraro 2008) are working on handling word order in VSMs.