Online Bayesian Models for Personal Analytics in Social Media Svitlana Volkova and Benjamin Van Durme Center for Language and Speech Processing, Johns Hopkins University, Human Language Technology Center of Excellence

Social Media Predictive Analytics Personalized, diverse and timely data Can reveal user interests, preferences and opinions Social Network Prediction App - DemographicsPro – WolphralAlpha Analytics –

User Attribute Prediction Task Political Preference Rao et al., 2010; Conover et al., 2011, Pennacchiotti and Popescu, 2011; Zamal et al., 2012; Cohen and Ruths, 2013; Volkova et. al, Communications Gender Garera and Yarowsky, 2009; Rao et al., 2010; Burger et al., 2011; Van Durme, 2012; Zamal et al., 2012; Bergsma and Van Durme, 2013 Age Rao et al., 2010; Zamal et al., 2012; Cohen and Ruth, 2013; Nguyen et al., 2011, 2013; Sap et al., 2014 … … … … AAAI 2015 Demo (joint work with Microsoft Research) Income, Education Level, Ethnicity, Life Satisfaction, Optimism, Personality, Showing Off, Self-Promoting

Outline I.Our Approach II.Dynamic (Streaming) Models III.Experimental Results IV.Practical Recommendations

Existing Approaches ~1K Tweets* …. … How long does it take for an average Twitter user to produce thousands of tweets? *Rao et al., 2010; Conover et al., 2011; Pennacchiotti and Popescu, 2011a; Burger et al., 2011; Zamal et al., 2012; Nguyen et al., 2013 Tweets as a document What if we want to make reliable predictions immediately after 10 tweets?

Attributed Social Networks *Conover et al., 2011; Pennacchiotti and Popescu, 2011a; Zamal et al., 2012; Volkova et al., 2014.

Our Approach Static (Batch) Predictions Streaming (Online) Inference Dynamic (Iterative) Learning and Prediction Offline training Offline predictions No or limited network information Offline training Online predictions in time (ACL’14) Exploring 6 types of neighborhoods ①Streaming nature of SM: dynamic training and prediction ②Network structure: joint user-neighbour streams ③Trade-off between prediction time vs. model quality Online predictions Relying on neighbors +Iterative re-training +Active learning +Interactive rationale annotation

Online Predictions: Iterative Bayesian Updates Time … ? ?

Iterative Batch Learning Time R D ? ? t1t1 … t1t1 Labeled Unlabeled t1t1 t1t1  Iterative Batch Retraining (IB)  Iterative Batch with Rationale Filtering (IBR) ? tmtm … tmtm t2t2 … t2t2 … tmtm t2t2 …

Rationales Rationales are explicitly highlighted ngrams in tweets that best justified why the annotators made their labeling decisions

Active Learning Labeled Unlabeled 1-Jan Feb Nov Dec-2011 Time … …  Active Without Oracle (AWOO)  Active With Rationale Filtering (AWR)  Active With Oracle (AWO)

Performance Metrics Accuracy over time: Find optimal models: – Data steam type (user, friend, user + friend) – Time (more correctly classified users faster) – Prediction quality (better accuracy over time)

Experimental Results: Data Stream and Prediction Time IBIBR AWOOAWR

Results: Iterative Batch Learning IB: higher recallIBR: higher precision Time: # correctly classified users increases over time IB faster, IBR slower Data stream selection: User + friend stream > user stream

Results: Active Learning AWOO: higher recallAWR: higher precision Time: Unlike IB/IBR models, AWOO/AWR models classify more users correctly faster (in Mar) but then plateaus

batch < active user + friend > user Results: Model Quality

Active with Oracle Annotations Oracle is 100% correct Thousands of tweets in training

Summary Active learning > iterative batch N, UN > U: “neighbors give you away” Higher confidence => higher precision, lower confidence => higher recall (as expected) Rationales significantly improve results

Practical Recommendations If you want to deliver ads fast but to be less confident in user attribute predictions: – use models with higher recall (AWOO, IB) – apply lower decision threshold e.g., 0.55 If you want to deliver ads to a true target crowd but latter in time: – use models with higher precision (AWR, IBR) – apply higher decision threshold e.g., 0.95 – models with rational filtering (IBR, AWR) require less computation (lower-dimensional feature vectors), are more accurate but annotations cost money (Mechanical Turk) For highly assortative attributes e.g., political preference use a joint user-neighbor stream

Applications Online targeted advertising Personalized marketing Large-scale real-time healthcare analytics Personalized recommendation systems and search Large-scale passive and real-time live polling Recruitment and human resource management Dating services

Applications (1) Online targeted advertising Targeting ads based on predicted user features Personalized marketing Detecting opinions users express about products or services within targeted populations Large-scale real-time healthcare analytics Identifying patterns of depression or mental illnesses within users of certain demographics Personalized recommendation systems and search

Applications (2) Recruitment and human resource management Estimating emotional stability and personality of the potential employees Measuring the overall well-being of the employees e.g., life satisfaction, happiness. Dating services Matching user profiles by comparing their personalities, interests and emotional tone in social media Large-scale passive and real-time live polling Mining political opinions, voting predictions for the groups of users with certain demographics

Thank you! Labeled Twitter network data for gender, age, political preference prediction: Interested in using our models for your research or collaboration: code and pre-trained models for inferring demographic attributes, personality and 6 Ekman’s emotions available on request: AAAI Technical Demo Inferring Latent User Properties from Texts Published in Social Media Wednesday, January 28 6:30 – 8:00 Zilker Ballroom I am on a job market. Hire me!

Cand-Centric Graph: Belief Updates ? … Time ? …

Cand-Centric: Prediction Time (1) User-Neighbor User Stream Dem Rep Prediction confidence: 0.95 vs Democrats are easier to predict than republicans Dem Rep Users classified correctly

Batch vs. Online Performance

Summary: Streaming (Online) Prediction Neighborhood content is useful * (homophily) Streaming models >> batch models Tweeting frequency matters a lot! Generalization of the classifiers depends on data sampling or annotation biases. *Pennacchiotti and Popescu, 2011a, 2001b; Conover et al., 2011a, 2001b; Golbeck et al., 2011; Zamal et al., 2012; Volkova et. al., 2014

Iterative Batch vs. Active w/o Oracle Predictions AWOO IB

Active with Oracle Learning