1. Modeling Information Seeking Behavior in Social Media Eugene Agichtein Intelligent Information Access Lab (IRLab)

2. Eugene Agichtein, Emory University, IR Lab 2 Qi Guo (3 rd year Phd) Ablimit Aji (2 nd year PhD) Modeling information seeking behavior Web search and social media search Text and data mining for medical informatics and public health In collaboration with: - Beth Buffalo (Neurology) - Charlie Clarke (Waterloo) - Ernie Garcia (Radiology) - Phil Wolff (Psychology) - Hongyuan Zha (GaTech) 1 st year graduate students: Julia Kiseleva, Dmitry Lagun, Qiaoling Liu, Wang Yu Yandong Liu (2 nd year Phd)

3. Online Behavior and Interactions Eugene Agichtein, Emory University, IR Lab 3 Information sharing: blogs, forums, discussions Search logs: queries, clicks Client-side behavior: Gaze tracking, mouse movement, scrolling

4. Research Overview Eugene Agichtein, Emory University, IR Lab 4 4 Information sharing Health Informatics Cognitive Diagnostics Intelligent search Discover Models of Behavior (machine learning/data mining)

5. Key Challenges for Web Search Query interpretation (infer intent) Ranking (high dimensionality) Evaluation (system improvement) Result presentation (information visualization) Eugene Agichtein, Emory University, IR Lab 5

6. Contextualized Intent Inference SERP text Mouse trajectory, hovering/dynamics Scrolling Clicks Eugene Agichtein, Emory University, IR Lab 6

7. Research Intent Eugene Agichtein, Emory University, IR Lab 7

8. Purchase Intent Eugene Agichtein, Emory University, IR Lab 8

9. Relationship between behavior and intent? Search intent is contextualized within a search session Implication 1: model session-level state Implication 2: improve detection based on client- side interactions Eugene Agichtein, Emory University, IR Lab 9

10. Model: Linear Chain CRF Eugene Agichtein, Emory University, IR Lab 10

11. Results: Ad Click Prediction 200%+ precision improvement (within mission) Eugene Agichtein, Emory University, IR Lab 11

12. Research Overview Eugene Agichtein, Emory University, IR Lab 12 Information sharing Health Informatics Cognitive Diagnostics Intelligent search Discover Models of Behavior (machine learning/data mining)

13. Finding Information Online (Revisited) 13 Next generation of search: Algorithmically-mediated information exchange CQA (collaborative question answering): Realistic information exchange Searching archives Train NLP, IR, QA systems Study of social behavior, norms Content quality, asker satisfaction Current and future work

14. Goal: Hybrid Human-Powered Search 14

15. Talk Outline Overview of the Emory IR Lab  Intent-centric Web Search  Classifying intent of a query  Contextualized search intent detection 15 Eugene Agichtein, Emory University, IR Lab

16. 16

17. (Text) Social Media Today Published: 4Gb/day Social Media: 10Gb/Day Technorati+Blogpulse 120M blogs 2M posts/day Twitter: since 11/07: 2M users 3M msgs/day Facebook/Myspace: 200-300M users Avg 19 m/day Yahoo Answers: 90M users, 20M questions, 400M answers [Data from Andrew Tomkins, SSM2008 Keynote] Yes, we could read your blog. Or, you could tell us about your day

18. 18

19. 19 Total time: 7-10 minutes, active “work”

20. Someone must know this…

21. 21 +1 minute

22. +7 hours: perfect answer

23. Update (2/15/2009) 23

24. 24 http://answers.yahoo.com/question/index;_ylt=3?qid=20071008115118AAh1HdO

25. 25

26. Finding Information Online (Revisited) 26 Next generation of search: Algorithmically-mediated information exchange CQA (collaborative question answering): Realistic information exchange Searching archives Train NLP, IR, QA systems Study of social behavior, norms Content quality, asker satisfaction Current and future work

27. (Some) Related Work Adamic et al., WWW 2007, WWW 2008: – Expertise sharing, network structure Elsas et al., SIGIR 2008: – Blog search Glance et al.: – Blog Pulse, popularity, information sharing Harper et al., CHI 2008, 2009: – Answer quality across multiple CQA sites Kraut et al.: – community participation Kumar et al., WWW 2004, KDD 2008, …: – Information diffusion in blogspace, network evolution SIGIR 2009 Workshop on Searching Social Media http://ir.mathcs.emory.edu/SSM2009/ 27

28. Finding High Quality Content in SM Well-written Interesting Relevant (answer) Factually correct Popular? Provocative? Useful? 28 As judged by professional editors E. Agichtein, C. Castillo, D. Donato, A. Gionis, and G. Mishne, Finding High Quality Content in Social Media, in WSDM 2008

29. Social Media Content Quality 29 E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne, Finding High Quality Content in Social Media, WSDM 2008 quality

30. 30 30

31. 31 How do Question and Answer Quality relate?

32. 32 32

33. 33 33

34. 34 34

35. 35 35

36. Community36

37. Link Analysis for Authority Estimation 37 Question 1 Question 2 Answer 5 Answer 1 Answer 2 Answer 4 Answer 3 User 1 User 2 User 3 User 6 User 4 User 5 Answer 6 Question 3 User 1 User 2 User 3 User 6 User 4 User 5 Hub (asker) Authority (answerer)

38. Qualitative Observations HITS effective   HITS ineffective 38

39. 39 39 Random forest classifier

40. Result 1: Identifying High Quality Questions 40

41. Top Features for Question Classification Asker popularity (“stars”) Punctuation density Question category Page views KL Divergence from reference LM 41

42. Identifying High Quality Answers 42

43. Top Features for Answer Classification Answer length Community ratings Answerer reputation Word overlap Kincaid readability score 43

44. Finding Information Online (Revisited) 44 Next generation of search: human-machine-human CQA: a case study in complex IR Content quality Asker satisfaction Understanding the interactions

45. Dimensions of “Quality” Well-written Interesting Relevant (answer) Factually correct Popular? Timely? Provocative? Useful? 45 As judged by the asker (or community)

46. Are Editor Labels “Meaningful” for CGC? Information seeking process: want to find useful information about topic with incomplete knowledge – N. Belkin: “Anomalous states of knowledge” Want to model directly if user found satisfactory information Specific (amenable) case: CQA

47. Yahoo! Answers: The Good News Active community of millions of users in many countries and languages Effective for subjective information needs – Great forum for socialization/chat Can be invaluable for hard-to-find information not available on the web 47

48. 48

49. Yahoo! Answers: The Bad News49 May have to wait a long time to get a satisfactory answer May never obtain a satisfying answer 1. FIFA World Cup 2. Optical 3. Poetry 4. Football (American) 5. Soccer 6. Medicine 7. Winter Sports 8. Special Education 9. General Health Care 10. Outdoor Recreation Time to close a question (hours)

50. Predicting Asker Satisfaction Given a question submitted by an asker in CQA, predict whether the user will be satisfied with the answers contributed by the community. – “Satisfied” : The asker has closed the question AND Selected the best answer AND Rated best answer >= 3 “stars” (# not important) – Else, “Unsatisfied 50 Yandong Liu Jiang Bian Y. Liu, J. Bian, and E. Agichtein, in SIGIR 2008

51. 51 ASP: Asker Satisfaction Prediction asker is satisfied asker is not satisfied Text Category Answerer History Asker History Answer Question Wikipedia News Classifier

52. 52 Experimental Setup: Data QuestionsAnswersAskersCategories% Satisfied 216,1701,963,615158,51510050.7% Crawled from Yahoo! Answers in early 2008 “Anonymized” dataset available at: http://ir.mathcs.emory.edu/shared/ 1/2009: Yahoo! Webscope : “Comprehensive” Answers dataset: ~5M questions & answers.

53. Satisfaction by Topic TopicQuestionsAnswersA per QSatisfiedAsker rating Time to close by asker 2006 FIFA World Cup 119435,659329.8655.4%2.6347 minutes Mental Health 15111597.6870.9%4.301.5 days Mathematics65123293.5844.5%4.4833 minutes Diet & Fitness 45024365.4168.4%4.301.5 days 53

54. 54 Satisfaction Prediction: Human Judges Truth: asker’s rating A random sample of 130 questions Researchers – Agreement: 0.82 F1: 0.45  2P*R/(P+R) Amazon Mechanical Turk – Five workers per question. – Agreement: 0.9 F1: 0.61 – Best when at least 4 out of 5 raters agree

55. Performance: ASP vs. Humans (F1, Satisfied) ClassifierWith TextWithout TextSelected Features ASP_SVM0.690.720.62 ASP_C4.50.750.760.77 ASP_RandomForest0.700.740.68 ASP_Boosting0.67 ASP_NB0.610.650.58 Best Human Perf0.61 Baseline (random)0.66 55 ASP is significantly more effective than humans Human F1 is lower than the random baseline!

56. Top Features by Information Gain 0.14 Q: Askers’ previous rating 0.14 Q: Average past rating by asker 0.10 UH: Member since (interval) 0.05 UH: Average # answers for by past Q 0.05 UH: Previous Q resolved for the asker 0.04 CA: Average asker rating for category 0.04 UH: Total number of answers received … 56

57. 57 “Offline” vs. “Online” Prediction Offline prediction (AFTER answers arrive) – All features( question, answer, asker & category) – F1: 0.77 Online prediction (BEFORE question posted) – NO answer features – Only asker history and question features (stars, #comments, sum of votes…) – F1: 0.74

58. Personalized Prediction of Satisfaction Same information != same usefulness for different searchers! Personalization vs. “Groupization”? 58 Y. Liu and E. Agichtein, You've Got Answers: Personalized Models for Predicting Success in Community Question Answering, ACL 2008

59. Example Personalized Models 59

60. Outline 60 Next generation of search: Algorithmically mediated information exchange CQA: a case study in complex IR Content quality Asker satisfaction

61. Current Work (in Progress) Partially supervised models of expertise (Bian et al., WWW 2009) Real-time CQA Sentiment, temporal sensitivity analysis Understanding Social Media dynamics

62. Answer Arrival 62

63. Exponential Decay Model [Lerman 2007]

64. Factors Influencing Dynamics

65. Example: Answer Arrival | Category

66. Subjectivity

67. Answer, Rating Arrival

68. Preliminary Results: Modeling SM Dynamics for Real-Time Classification Adapt SM dynamics models to classification e.g.: predict ratings  feature value:

69. Outline 69 Next generation of search: Algorithmically mediated information exchange CQA: a case study in complex IR Content quality Asker satisfaction Understanding social media dynamics

70. Question Urgency Eugene Agichtein, Emory University, IR Lab 70 Problem – a growing volume of questions competing for visibility Time-sensitive (urgent) questions pushed out by newer questions Delayed responses may become useless to seeker – wastes site resources and responders’ time

71. Goal: Query Processing over Web and Social Systems 71

72. Takeaways Robust machine learning over behavior data  system improvements, insights into behavior Contextualized models for NLP and text mining  system improvements, insights into interactions Mining social media: potential for transformative impact for IR, sociology, psychology, medical informatics, public health, … 72

73. References Modeling web search behavior [SIGIR 2006, 2007] Estimating content quality [WSDM 2008] Estimating contributor authority [CIKM 2007] Searching CQA archives [WWW 2008, WWW 2009] Inferring asker intent [EMNLP 2008] Predicting satisfaction [SIGIR 2008, ACL 2008, TKDE] Coping with spam [AIRWeb 2008] More information, datasets, papers, slides: http://www.mathcs.emory.edu/~eugene/

74. Thank you! Yandex (for hosting my visit) Eugene Agichtein, Emory University, IR Lab 74 Supported by: