1. Joint Models of Disagreement and Stance in Online Debate Dhanya Sridhar, James Foulds, Bert Huang, Lise Getoor, Marilyn Walker University of California, Santa Cruz Virginia Institute of Technology 1 121 11 1 2

2. Social media sites for debating issues Valuable resources for: –Argumentation –Dialogue –Sentiment –Opinion mining 2 Online Debate Forums

3. CreateDebate.org 3

4. 4 Debate topic

5. CreateDebate.org 5 Debate topic Posts

6. CreateDebate.org 6 Debate topic Posts Replies

7. CreateDebate.org 7 Debate topic Posts Replies Reply polarity

8. 4Forums.com 8

9. 9 Quotation

10. Graph of posts: tree structure Online Debate Forums 10 Graph of users: loopy structure = reply link

11. Graph of posts: tree structure Online Debate Forums 11 Graph of users: loopy structure

12. Zoom in on an Example 12

13. Zoom in on an Example 13

14. I believe Obama is likely the worst! He’s been infinitely more effective than Bush! Zoom in on an Example 14

15. I believe Obama is likely the worst! He’s been infinitely more effective than Bush! Anti Obama Pro Obama Task: Stance Classification 15 Useful for advocacy and get-out-the-vote campaigns

16. I believe Obama is likely the worst! He’s been infinitely more effective than Bush! Anti Obama Pro Obama Task: Stance Classification 16 Study argumentation and dialogue Posts express stance

17. I believe Obama is likely the worst! He’s been infinitely more effective than Bush! Anti Obama Pro Obama Task: Disagreement Classification 17 Study argumentation and dialogue Disagree on stance

18. I believe Obama is likely the worst! He’s been infinitely more effective than Bush! Anti Obama Pro Obama Task: Disagreement Classification 18 Study argumentation and dialogue Disagree on stance Posts express disagreement

19. Stance Classification Targets 19 Stance Author-level Post-level Disagreement Author-level Post-level Textual

20. Stance Disagrees Classification Targets 20 Stance Author-level Post-level Disagreement Author-level Post-level Textual

21. Stance Disagrees Classification Targets 21 Stance Author-level Post-level Disagreement Author-level Post-level Textual

22. Related Work Walker et. al (2012), Decision Support Sciences State-of-the-art local stance classifier using linguistic features Walker et. al (2012), NAACL Collective stance classification using MaxCut over rebuttal links Hasan and Ng. (2013), EMNLP Collective stance sequence labeling with CRF; benefits of user stance consistency constraint Abbott et. al (2012), ACL Local disagreement classifier using linguistic features Burfoot et. al (2011), ACL Joint classification of stance and reply polarity in Congressional debates 22

23. Related Work Walker et. al (2012), Decision Support Sciences State-of-the-art local stance classifier using linguistic features Walker et. al (2012), NAACL Collective stance classification using MaxCut over rebuttal links Hasan and Ng. (2013), EMNLP Collective stance sequence labeling with CRF; benefits of user stance consistency constraint Abbott et. al (2012), ACL Local disagreement classifier using linguistic features Burfoot et. al (2011), ACL Joint classification of stance and reply polarity in Congressional debates 23

24. Related Work Walker et. al (2012), Decision Support Sciences State-of-the-art local stance classifier using linguistic features Walker et. al (2012), NAACL Collective stance classification using MaxCut over rebuttal links Hasan and Ng. (2013), EMNLP Collective stance sequence labeling with CRF; benefits of user stance consistency constraint Abbott et. al (2012), ACL Local disagreement classifier using linguistic features Burfoot et. al (2011), ACL Joint classification of stance and reply polarity in Congressional debates 24

25. Related Work Walker et. al (2012), Decision Support Sciences State-of-the-art local stance classifier using linguistic features Walker et. al (2012), NAACL Collective stance classification using MaxCut over rebuttal links Hasan and Ng. (2013), EMNLP Collective stance sequence labeling with CRF; benefits of user stance consistency constraint Abbott et. al (2012), ACL Local disagreement classifier using linguistic features Burfoot et. al (2011), ACL Joint classification of stance and reply polarity in Congressional debates 25

26. Related Work Walker et. al (2012), Decision Support Sciences State-of-the-art local stance classifier using linguistic features Walker et. al (2012), NAACL Collective stance classification using MaxCut over rebuttal links Hasan and Ng. (2013), EMNLP Collective stance sequence labeling with CRF; benefits of user stance consistency constraint Abbott et. al (2012), ACL Local disagreement classifier using linguistic features Burfoot et. al (2011), ACL Joint classification of stance and reply polarity in Congressional debates 26

27. Stance Classification: “Teach the Controversy” Previous work employs many modeling strategies How best to model stance in online debate? Answers may be different to Congressional debates –Links have different semantics –Posts much shorter than speeches. Many posts per author –Dialogue is informal 27

28. Stance Classification: “Teach the Controversy” Previous work employs many modeling strategies How best to model stance in online debate? Answers may be different to Congressional debates –Links have different semantics –Posts much shorter than speeches. Many posts per author –Dialogue is informal 28

29. Stance Modeling at author-level or post-level? 29 [Hasan and Ng 2013] [Other Related Work] Modeling Question 1)

30. Stance Modeling at author-level or post-level? 30 [Hasan and Ng 2013] [Other Related Work] Modeling Question 1)

31. Stance Modeling Question 2) 31 [Walker et al. 2012, Hasan and Ng 2013] Collective classification vs. local classification?

32. Stance Modeling Question 2) 32 [Walker et al. 2012, Hasan and Ng 2013 ] [Walker et al. 2012] Collective classification vs. local classification?

33. Stance Disagrees 33 Jointly model disagreement together with stance? [Abbott et. al 2012 - Linguistic Features], [Burfoot et. al 2011 for Congressional Debates] Modeling Question 3)

34. Stance Disagrees 34 Jointly model disagreement together with stance? [Abbott et. al 2012 - Linguistic Features], [Burfoot et. al 2011 for Congressional Debates] Modeling Question 3) Stance Disagrees Stance

35. Our Contributions A unified framework to explore multiple models Fast, highly scalable inference –Large post-level graphs –Loopy author-level graphs Systematic study of modeling options 35

36. Our Contributions A unified framework to explore multiple models Fast, highly scalable inference –Large post-level graphs –Loopy author-level graphs Systematic study of modeling options 36

37. Our Contributions A unified framework to explore multiple models Fast, highly scalable inference –Large post-level graphs –Loopy author-level graphs Systematic study of modeling options –Modeling recommendations 37

38. Author Post Local Collective Joint Author Local Author Coll. Author Joint Post Local Post Joint Post Coll. Modeling Granularity Statistical Models All Combinations of Models 38

39. Probabilistic Soft Logic (PSL) Templating language for highly scalable graphical model called Hinge-loss Markov Random Fields 39 Bach et. al (2015), ArXiV Open source software: https://psl.umiacs.umd.edu 5.0: Disagrees(A1, A2) ^ Pro(A1)  ~Pro(A2) Rule Weight

40. Probabilistic Soft Logic (PSL) Templating language for highly scalable graphical model called Hinge-loss Markov Random Fields 40 5.0: Disagrees(A1, A2) ^ Pro(A1)  ~Pro(A2) Rule Weight Predicates are continuous Random Variables!

41. Probabilistic Soft Logic (PSL) Templating language for highly scalable graphical model called Hinge-loss Markov Random Fields 41 5.0: Disagrees(A1, A2) ^ Pro(A1)  ~Pro(A2) Rule Weight Predicates are continuous Random Variables! Relaxations of Logical Operators

42. Probabilistic Soft Logic (PSL) Rules instantiated with variables from real network 42 5.0: Disagrees(A1, A2) ^ Pro(A1)  ~Pro(A2) Stance Disagrees

43. Probabilistic Soft Logic (PSL) Rules instantiated with variables from real network 43 5.0: Disagrees(, ) ^ Pro( )  ~Pro( ) … Continuous Random Variables!

44. Hinge-loss MRFs Over Continuous Variables Bach et al. NIPS 12, Bach et al. UAI 13 44 Bach et al. (2015), ArXiV Conditional random field over continuous RVs in [0,1]

45. Hinge-loss MRFs Over Continuous Variables Bach et al. NIPS 12, Bach et al. UAI 13 45 Conditional random field over continuous RVs in [0,1] Feature function for each instantiated rule 5.0: Disagrees(, ) ^ Pro( )  ~Pro( )

46. Hinge-loss MRFs Over Continuous Variables Bach et al. NIPS 12, Bach et al. UAI 13 46 Conditional random field over continuous RVs in [0,1] Feature functions are hinge-loss functions

47. Hinge-loss MRFs Over Continuous Variables Bach et al. NIPS 12, Bach et al. UAI 13 47 Encodes distance to satisfaction of each instantiated rule Linear function

48. Fast Inference in Hinge-loss MRFs Bach et al. NIPS 12, Bach et al. UAI 13 48 Convex, continuous inference objective… Convex optimization! Solved using efficient, parallelizable algorithm: Alternating Direction Method of Multipliers (ADMM)

49. Unigrams, Bigrams, Lengths, Initial n-grams, Repeated Punctuation Obama Bush believe Constructing Local Predictors 49 Bag-of-words

50. Unigrams, Bigrams, Lengths, Initial n-grams, Repeated Punctuation Logistic Regression Obama Bush believe Constructing Local Predictors 50 Pro Not Pro Bag-of-words Training Labels

51. Unigrams, Bigrams, Lengths, Initial n-grams, Repeated Punctuation Logistic Regression Observed Prediction Probabilities Obama Bush believe Constructing Local Predictors 51 Pro Not Pro Bag-of-words Training Labels LocalPro: 0.8 LocalPro: 0.1

52. PSL Rules Shared by All Models 52 Stance LocalPro: 0.8 LocalPro: 0.1

53. PSL Rules for Simple Collective Models 53 Stance Disagrees: 1.0 LocalPro: 0.8 LocalPro: 0.1

54. Joint Disagreement Models 54 Stance Disagrees LocalPro: 0.8 LocalPro: 0.1 LocalDis: 0.5

55. Joint Disagreement Models 55 Stance Disagrees LocalPro: 0.8 LocalPro: 0.1 LocalDis: 0.5

56. Details4Forums.comCreateDebate.org TopicsAbortion, Gay Marriage, Evolution, Gun Control Abortion, Gay Rights, Obama, Marijuana Avg. Users/Topic 336311 Avg. Posts/User 194 Evaluation - Datasets 56

57. Evaluation Settings Prediction Tasks: Author Stance, Post Stance 57

58. Evaluation Settings Prediction Tasks: Author Stance, Post Stance 58 Ground Truth for CreateDebate.org: Stance Majority

59. Evaluation Settings Prediction Tasks: Author Stance, Post Stance 59 Ground Truth for CreateDebate.org: Stance Majority Ground Truth for 4Forums: Stance

60. 60 Post Local Post Coll. Post Joint Author Local Author Coll. Author Joint Accuracy Author Stance Prediction – CreateDebate.org

61. 61 Post Local Post Coll. Post Joint Author Local Author Coll. Author Joint Accuracy Author Stance Prediction – CreateDebate.org Post < Author

62. 62 Post Local Post Coll. Post Joint Author Local Author Coll. Author Joint Accuracy Author Stance Prediction – CreateDebate.org Post < Author Author-Joint Model is best

63. 63 Accuracy Post Local Post Coll. Post Joint Author Local Author Coll. Author Joint Post Stance Prediction – CreateDebate.org Post < Author (still!) Author-Joint Model still best!

64. 64 Post Local Post Coll. Post Joint Author Local Author Coll. Author Joint Accuracy Author Stance Prediction – CreateDebate.org Local < Collective < Joint

65. 65 Post Local Post Coll. Post Joint Author Local Author Coll. Author Joint Accuracy Naïve collective harmful at author level! Author Stance Prediction – 4Forums.com

66. 66 Post Local Post Coll. Post Joint Author Local Author Coll. Author Joint Accuracy Author Stance Prediction – 4Forums.com Naïve collective harmful at author level!

67. Details4Forums.comCreateDebate.org % Opposite Stance Posts 71.673.9 % Opposite Stance Authors 52.068.9 Explanation for Naïve Collective’s Performance 67

68. Details4Forums.comCreateDebate.org % Opposite Stance Posts 71.673.9 % Opposite Stance Authors 52.068.9 Explanation for Naïve Collective’s Performance 68 Naïve collective assumption mostly true for posts

69. Details4Forums.comCreateDebate.org % Opposite Stance Posts 71.673.9 % Opposite Stance Authors 52.0 68.9 Explanation for Naïve Collective’s Performance 69 Naïve collective assumption mostly true for posts Assumption doesn’t hold at author level!

70. 70 I agree with everything except the last part. Safe gun storage is very important… I agree with everything except the last part. Safe gun storage is very important… I can agree with this. And in case it seemed otherwise, I know full well how to store guns safely… My point was that I don’t like the idea of such a law… I can agree with this. And in case it seemed otherwise, I know full well how to store guns safely… My point was that I don’t like the idea of such a law… Benefit of Disagreement Prediction Agree Anti Gun Control

71. 71 I agree with everything except the last part. Safe gun storage is very important… I agree with everything except the last part. Safe gun storage is very important… I can agree with this. And in case it seemed otherwise, I know full well how to store guns safely… My point was that I don’t like the idea of such a law… I can agree with this. And in case it seemed otherwise, I know full well how to store guns safely… My point was that I don’t like the idea of such a law… Benefit of Disagreement Prediction Agree Anti Gun Control

72. Summary Unified modeling framework for efficiently, systematically exploring all modeling choices Author-level joint disagreement and stance model best, even for post-level prediction Disagreement model can be vital when modeling at author level 72

73. Summary Unified modeling framework for efficiently, systematically exploring all modeling choices Author-level joint disagreement and stance model best, even for post-level prediction Disagreement model can be vital when modeling at author level 73

74. Summary Unified modeling framework for efficiently, systematically exploring all modeling choices Author-level joint disagreement and stance model best, even for post-level prediction Disagreement model can be vital when modeling at author level 74

75. Summary Unified modeling framework for efficiently, systematically exploring all modeling choices Author-level joint disagreement and stance model best, even for post-level prediction Disagreement model can be vital when modeling at author level 75 Thank you for your attention!