+ Improving Vector Space Word Representations Using Multilingual Correlation Manaal Faruqui and Chris Dyer Language Technologies Institute Carnegie Mellon University

+ Distributional Semantics “You shall know a word by the company it keeps” (Harris 1954; Firth, 1957) …I will take what is mine with fire and blood… …the end battle would be between fire and ice… …My dragons are large and can breathe fire now… …flame is the visible portion of a fire… …take place whereby fires can sustain their own heat…

+ Translational Semantics What other Information? (Bannard & Callison-Burch, 2005) तीन सौ से अधिक लोगों को बैठाने वाला वायुयान … That plane can seat more than 300 people Russian airplanes are huge रूसी वायुयान बहुत बड़े हैं Multilingual Information! plane ≅ airplane

+ Outline Distributional Semantics Monolingual context Translational Semantics Multilingual context Better Semantic Representations Using Distributional + Translational semantics

+ Word Vector Representations How to encode such co-occurrences? daynight…cold sleep0102 winter3350 … the10129 contexts words

+ Word Vector Representation Latent Semantic Analysis (Deerwester et al., 1990) Singular Value Decomposition words context words

+ Multilingual Information English German French Spanish dragon Drache dragon dragón Problem ? = Append

+ Multilingual Information Vector Size Increases Idiosyncratic Info. What if word is OOV ? Disadvantages of Vector Concatenation ✗ ?

+ Multilingual Information …I will take what is mine with fire and blood… …the end battle would be between fire and ice… …My dragons are large and can breathe fire now… So, what can we do?... Das Ende der Schlacht würde zwischen Feuer und Eis gesehen ist Feuer eine Oxidationsreaktion mit Das Licht des Feuers ist eine physikalische Erscheinung… Two Views: Canonical Correlation Analysis !

+ Canonical Correlation Analysis (CCA) Project two sets of vectors (equal cardinality) in a space where they are maximally correlated ΩΘ Convex Optimization Problem with Exact Solution ! ΩΘ ≅ CCA

+ Canonical Correlation Analysis (CCA) k = min(r( Ω ), r( Θ )) W V X Y × × n2n2 d1d1 k n1n1 d2d2 d2d2 k d1d1 X”X” Y”Y” k k n2n2 n1n1 X ” and Y ” are now maximally correlated ! W, V = CCA( Ω, Θ )

+ Canonical Correlation Analysis (CCA) Vector Size Increases, Doesn’t increase Problems Addressed? Idiosyncratic Information, Lets you choose! What if word is OOV?, Projection vectors for everyone!

+ Canonical Correlation Analysis (CCA) The vocabularies cant be of equal size ! Ok, but equal cardinality sets Ω & Θ ? Get word alignments from a parallel corpus Preserve only words in the original vocabulary For every word in English, select the best foreign word

+ Experimental Setup LSA Word Vector Learning Monolingua l Data EnglishGermanFrenchSpanish News CorpusWMT-2011 WMT WMT-2011 Tokens360,000,000290,000,000263,000,000164,000,000 Types180,000294,000137,000145,000 Tokenizer and Lowercasing: WMT scripts

+ Experimental Setup LSA Word Vector Learning Parallel Data De-EnFr-EnEs-En News Comm + Europarl WMT Tokens128,000,000138,000,000134,000,000 Word pairs37,00038,000 Word Alignment Tool: fast_align (Dyer et al, 2013)

+ Experimental Setup LSA Word Vector Learning Corpus Preprocessing...hello… …hello… …hello… …hello… …hello… Context : 23.45, 21 st, , 0.5e10 NUM anchfgugsjh, wekjfbg, bhguyq UNK

+ Experimental Setup Word Similarity Evaluation WS-353 (Finkelstein et al, 2001) WS-353-SIM (Agirre et al, 2009) WS-353-REL (Agirre et al, 2009) RG-65 (Rubenstein and Goodenough, 1965) MC-30 (Miller and Charles, 1991) MTurk-287 (Radinsky et al, 2011) Word Relation Evaluation Semantic Relations (Mikolov et al, 2013) Syntactic Relations (Mikolov et al, 2013) Evaluation Benchmarks

+ Experimental Setup Monolingual Vector Length: 80 Multilingual Vector Length: ? Multilingual Vector Learning The length in projected space can be chosen: ‘k’ Choose the best value of ‘k’ for WS-353 k ε [0.1, 0.2, …, 1.0]

+ Experimental Setup Multilingual Vector Learning Performance on WS-353; k = 0.6 Spearman’s correlation Dimensions

+ Experimental Setup Multilingual Vector Learning Spearman’s correlation

+ Experimental Setup Multilingual Vector Learning Accuracy

+ Experimental Setup RNNLM (Mikolov et al, 2011) Predict next word given the history Neural language model Recurrent hidden layer connections Skip-Gram, word2vec (Mikolov et al, 2013) Predict context given the word Removes hidden layer Vocabulary represented in Huffman coding Multilingual Vectors: Neural Networks

+ Experimental Setup Multilingual Vector Learning RNNLM Skip-Gram

+ Experimental Setup Multilingual Vectors: Scaling Spearman’s correlation on WS-353

+ Experimental Setup Multilingual Vectors: Qualitative Analysis Antonyms and Synonyms of “Beautiful”: Monolingual Setting t-SNE tool (van der Maaten and Hinton, 2008)

+ Experimental Setup Multilingual Vectors: Qualitative Analysis Antonyms and Synonyms of “Beautiful”: Multilingual Setting t-SNE tool (van der Maaten and Hinton, 2008)

+ Conclusion CCA: Easy to use tool in MATLAB Take vectors from two languages and improve them. Multilingual Information is Important Even if the problems are inherently monolingual. More Effective for Distributional Vectors Semantics generalizes better than Syntax. Vectors available at:

+ Related Work Document representation Vinokourov et al, 2002, Platt et al, 2010 Synonymy and Paraphrasing Bannard and Burch, 2005, Ganitkevitch et al, 2013 Bilingual lexicon induction Haghighi et al, 2008 Vulic and Moens, 2013 Bilingual word vectors Klementiev et al 2012 Zou et al, 2013 Translation Models Kalbrenner & Blunsom, 2013 Compositional Semantics Hermann & Blunsom, 2014

+ Thanks! Visit us at ACL-demo: wordvectors.org