Socialized Word Embeddings Ziqian Zeng1, Yichun Yin1,2, Yangqiu Song1 and Ming Zhang2 The Hong Kong University of Science and Technology1, Peking University2

Motivation Facts Everyone has his/her own personal characteristics of language use.

Motivation Facts Linguistic homophily: socially connected individuals tend to use language in similar ways. 1 2 Frequently Used Words Language feature tdd, mvc, linq acronyms anipals, pawsome, furever animal based puns kradam, glambert, glamily puns around pop star Adam Lambert Yi Yang et al. Overcoming Language Variation in Sentiment Analysis with Social Attention, 2016. Table is from Bryden John et al, Word usage mirrors community structure in the online social network Twitter, 2013 with partial deletion.

Motivation Facts Everyone has his/her own personal characteristics of language use. Linguistic homophily: socially connected individuals tend to use language in similar ways. 1 Goal Develop a word embeddings algorithm which can consider the facts. Yi Yang et al. Overcoming Language Variation in Sentiment Analysis with Social Attention, 2016.

CBOW Figure 1: Illustration of CBOW (1) in the paper, we take CBOW as an example to introduce our algorithm. Figure 1: Illustration of CBOW Mikolov Tomas et al. Efficient estimation of word representations in vector space. 2013

Socialized Word Embeddings Figure 2: Illustration of Socialized Word Embeddings

Socialized Word Embeddings Personalized Fact Everyone has his/her own personal characteristics of language use. Figure 2: Illustration of Socialized Word Embeddings

Socialized Word Embeddings Fact Linguistic homophily: socially connected individuals tend to use language in similar ways. Figure 2: Illustration of Socialized Word Embeddings

Personalization Notations Personalized CBOW users: A word ’s context , where is the half window size. A corpus for user . Global word vector: ; Local user vector: . Vector representation of a word for user : Personalized CBOW (2)

Socialization Notations Socialized Regularization , where is the number of friends of . Socialized Regularization (3)

Socialization Personalized CBOW Socialized Regularization (2) Socialized Regularization (3) Socialized Word Embeddings Trade-off Parameter (4) Constraint

Experiments Dataset – Yelp Challenge Yelp Challenge Datasets contain billions of reviews and ratings for various businesses.

Experiments Perplexity We reported: Perplexity is used to evaluate how good a model is to predict the current word based on several previous words. We reported: Perplexity trend with varied ( -norm constraint for user vectors) and fixed λ (socialized regularization parameter). Better

Experiments Perplexity trend with varied λ and fixed Better

Experiments SVM Sentiment Classification + average review user classifier rating Figure 5. Illustration of the procedure of sentiment classification

Experiments Sentiment Classification – Head & Tail Statistics Train on only head users or tail users. Select the users who contributed half of the total reviews as head users, and the other users are left as tail users. Statistics

Experiments Better

Experiments Better

Experiments User Vector as Attention User attention vectors can improve sentiment classification. User vectors as fixed attention vectors. Comparison with the baseline without attention, and the upper bound with trainable attention. Figure 8: The architecture of User Product Attention based Neural Sentiment Classification model.1 Figure 8 is form Chen Huimin et al. Neural sentiment classification with user and product attention. 2016

Experiments Supervised Better Unsupervised Figure 9: Comparison of our model and other baseline methods on user attention based deep learning for sentiment analysis.

Conclusion Representation: global word vector + local user vector Socialized regularization: friends’ user vectors should be similar. Thank You Figure 2. Illustration of Socialized Word Embeddings