Carolyn Penstein Rosé Language Technologies Institute and Human-Computer Interaction Institute With funding from the National Science Foundation and the Office of Naval Research

Acknowledgements Dong Nguyen Elijah Mayfield Hua Ai Rohit Kumar Iris Howley Nguyen, D., Mayfield, E., & Rosé, C. P. (2010). An analysis of perspectives in interactive settings, in Proceedings of the KDD Workshop on Social Media Analytics. Kumar, R. & Rosé, C. P. (2010). Engaging learning groups using Social Interaction Strategies, In Proceedings of the North American Chapter of the Association for Computational Linguistics. Howley, I., Mayfield, E. & Rosé, C. P. (to appear). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc. Ai, H., Kumar, R., Nguyen, D., Nagasunder, A., Rosé, C. P. (2010). Exploring the Effectiveness of Social Capabilities and Goal Alignment in Computer Supported Collaborative Learning, in Proceedings of Intelligent Tutoring Systems. Dong Nguyen Elijah Mayfield Hua Ai Rohit Kumar Iris Howley For more information about my group:

Outline Motivation from Opinion Mining Theoretical framework from Rhetoric and Discourse Analysis Study one: Political bias in a political discussion forum Study two: Goal orientation in chat based design discussions Current Directions

Outline Motivation from Opinion Mining Theoretical framework from Rhetoric and Discourse Analysis Study one: Political bias in a political discussion forum Study two: Goal orientation in chat based design discussions Current Directions

Typical paradigm for sentiment analysis of product reviews: Make a prediction based on text of single reviews taken out of context Some evidence of group effects in product review blogs based on numerical ratings (Wu et al., 2008) KEY ASSUMPTION: language is a reflection of the speaker’s perspective Are product reviews conversational? KEY ASSUMPTION: language is a reflection of the speaker’s perspective

Work towards weakening the assumption Sometimes use syntactic cues to reverse polarity on some terms (Somasunderan & Wiebe, 2009; Wijaya & Bressan, 2008) Factoring out the effect of context rather than modeling it Aggregation over all reviews posted by the same individual Taking opinions of similar individuals into account (e.g., collaborative filtering) Falls short of modeling conversational aspects of product reviews KEY ASSUMPTION: language is a reflection of the speaker’s perspective Are product reviews conversational? KEY ASSUMPTION: language is a reflection of the speaker’s perspective

Are product reviews conversational? “After many MANY weeks of research, gathering information from several sites, reviews etc I decided that the Britax Boulevard was definitely the safest bet available on the market. The things that sold me: All the safety gadgets that other seats don't have like the side impact wings, the HUGS system, the LATCH system and 5 point harness and also the fact that it lasts up to 29Kg. “

Are product reviews conversational? “I did most of my research on the net, picking my top 3 choices I went and had a look at them in the shops. I looked at one the Graco Comfort Sport, the Britax Boulevard and the Decathlon and Marathon seats. By far it seems that Britax have the upper hand safely wise on the market, many professional reviews and crash tests agree on this so Britax was the clear choice for us. “

Are product reviews conversational? “I have the seat front facing in my Camry (2007) I worried about the size of the chair from reading other reviews but that is NO problem in my car, my son has plenty of leg room and can see perfectly out the window.”

Are product reviews conversational?

Outline Motivation from Opinion Mining Theoretical framework from Rhetoric and Discourse Analysis Study one: Political bias in a political discussion forum Study two: Goal orientation in chat based design discussions Current Directions

Discourse and Identity Identity is reflected in the way we present ourselves in conversational interactions Reflects who we are, how we think, and where we belong Also reflects how we think of our audience Examples Regional dialect: shows my identification with where I am from, but also shows I am comfortable letting you identify me that way Jargon and technical terms: shows my identification with a work community, but also shows I expect you to be able to relate to that part of my life Level of formality: shows where we stand in relation to one another Explicitness in reference: shows whether I am treating you like an insider or an outsider

Discourse and Identity Lave & Wenger, 1991 Discourse is text above the clause level (Martin & Rose, 2007) A Discourse is an ongoing conversation [type] Socialization is the process of joining a Discourse (Lave & Wenger, 1991; Sfard, 2010) We join Discourses that match our core identity (de Fina, Schiffrin, & Bamberg, 2006) In moving from the periphery to the core of a Discourse community, we sound more and more like the community (Arguello et al., 2006) A discourse is one instance of it [token] All discourses contain echoes of previous discourses (Bakhtin, 1983) Lakoff & Johnson, 1980

Metaphors Structure our Experience We describe arguments using terms related to war Using a typical war ‘script’ to structure a story about an argument We orient towards arguments as though they were wars Our conversational partner is our opponent We may feel that we won or lost We may feel wounded as a result

Discourses, Frames, and Metaphors Frame: A portion of a discourse belonging to distinct Discourse Metaphor : One linguistic device that can be used to define a set of discourse practices that constitute a frame Topic models: a technical approach that makes sense for identifying frames within a discourse A discourse could be drawn from a mixture of Discourses Within the same conversation, we may wear a variety of “hats” E.g., the same discourse with a co-worker may contain exchanges pertaining to our relationship as colleagues and others to our relationship as friends

Model of Communication from Rhetoric Implied author: Communication style is a projection of identity Impression management, not necessarily the ground truth Implied reader: What we assume about who is listening Real assumptions, possibly incorrect What we want recipients or overhearers to think are our assumptions Reader: may or may not understand the text the way it was intended Author Implied Author Implied Reader Text Effect Reader

Engagement: Social positioning in conversational style The message: Most contributions express some content Implied author: How I phrase it says something about my stance with respect to that content Implied reader: Also says something about what I assume is your stance and my stance in relation to you Reader: The hearer may respond either to the message or its positioning Author Implied Author Implied Reader Text Effect Reader

Engagement: Social positioning in conversational style Speaker 1: I want chocolate for dessert. Speaker 2: [you can’t have chocolate] Options with different implications about author and reader You can’t have chocolate. You’re allergic to chocolate. You’re allergic to chocolate, so eating it would be a bad idea. Your mom said you’re allergic to chocolate Having chocolate might be a poor choice for you. Having chocolate might be a poor choice for you for a great number of reasons. Author Implied Author Implied Reader Text Effect Reader

Even Scientific Writing is Social and Conversational Implied author: Rhetorical style in academic writing gives an impression of who we are as researchers Implied reader: targeting writing to community standards Abstracts and literature reviews position us in research community Reader: research papers teach us both about the content of our field and its politics Author Implied Author Implied Reader Text Effect Reader

Outline Motivation from Opinion Mining Theoretical framework from Rhetoric and Discourse Analysis Study one: Political bias in a political discussion forum Study two: Goal orientation in chat based design discussions Current Directions

Bias Estimation Start with LDA model of politics dataset with 15 topics Then separate the texts into two collections, one left affiliated, and one right affiliated We then have a Left model and a Right model We can then compute a rank for each word w in each topic t in each model Intuition: a word is more distinguishing for a particular point of view if it has a high probability within the associated model and a low probability in the opposite model Bias(w,t) = log(rank right (w,t) + 1) – log(rank left (w,t) + 1) The bias of a text is the average bias over the terms within the text Left scores positive, right scores negative

Qualitative Analysis Terror Language (Right): evokes emotional response to thread of attack. Define target as evil and as a threat. Provokes a defensive posture. Imperialist rhetoric (Right): racial prejudice, attitude of superiority. Web of concern (Left): focus on opposition as individuals with a culture and history, concern for wellbeing of all people, focus on potential negative effects of war

Quantitative Analysis Right Bias Left Bias Score of poster Score of quoted message Score of full post Score of words that appear in both messages Score of words that appear only in quoted message Score of words that appear only in the post

Investigation of Quoting behavior Negative correlation between words only in quoted message and words only in post (r=-0.1, p < 0.05) Positive correlation between score quoted words and score of the whole post (r=0.18, p < 0.02) Score of words only in post are significantly more reflective of the affiliation of the poster than that of the author of the quoted message Similar result with score of words only in quote with affiliation of author of quoted message

Investigation of Quoting behavior

Which words are quoted? by pointing out the inflation of Saddam’s body count by neocons in an effort to further vilify him and thus further justify our invasion we are not DEFENDING saddam....just pointing out how neocons rarely let facts get in the way of a good war. So wait, how many do you think Saddam killed or oppressed? You’re trying to make him look better than he actually was. You’re the one inflating the casualties we’ve caused! Seriously, what estimates (with a link) are there that we’ve killed over 100,000 civilians. Not some crack pot geocities page either.

Thread level analysis Effect of initial post Correlation between score thread (without first post) and first post = (p<0.01) Effect of Prior posts Aggregate score of previous posts Difference in score of current post and average score of user Small correlation (r=0.133, p < 0.01) indicating that users talk more left than they usually do when previous posts are left and similarly for right.

Overview of Findings Quotes from opposite point of view include the words that are less strongly associated with the opposite perspective Because of quotes, displayed bias shifts towards the bias of the person to whom the message is directed Personal bias of the speaker is most strongly represented by non-quoted portions of text The effect of a post extends past just the immediate response

Outline Motivation from Opinion Mining Theoretical framework from Rhetoric and Discourse Analysis Study one: Political bias in a political discussion forum Study two: Goal orientation in chat based design discussions Current Directions

Collaborative Design Task Goal: Design a power plant Competing Student Goals: Power: Design a power plant that achieves maximum power output Green: Design a power plant that has the minimum impact on the environment Competing goals encourages deeper discussion Exploration of design space Explicit articulation of reasoning 30 > Experimental Design >Task

31 Classroom Study Chat room style interaction ConcertChat 106 students CMU undergrads in ME Classroom session During the semester Instruction  Pretest  Session  Posttest  Questionnaire 31 > Experimental Design >Study Procedure

Experimental Design Hypotheses Students will be more engaged with agents displaying social behaviors Students will be sensitive to tutor goal orientation Interaction effect Frequent Green Infrequent Green Frequent Neutral None Power None Neutral Frequent Power Infrequent Neutral Infrequent Power None Green Social Behavior –Frequent, Infrequent, None Goal Alignment –Green, Power, Neutral 32 > Experimental Design >Manipulations

An Example of Displaying Bias 33 > Experimental Design >Agent Design Green Bias Green: What is bad about increasing heat input to the cycle is that more waste heat is rejected to the environment. Neutral and Power: Increasing heat input to the cycle increases waste heat rejected to the environment. Power Bias: Power: What is good about increasing heat input to the cycle is that more power output is produced. Neutral and Green: Increasing heat input to the cycle increases power output produced.

34 Example of Social Behaviors 34 > Experimental Design >Agent Design 1. Showing Solidarity: Raises other's status, gives help, reward 2. Showing Tension Release: Jokes, laughs, shows satisfaction 3. Agreeing: Shows passive acceptance, understands, concurs, complies Tutor: Let’s Introduce ourselves. My name is Avis. Tutor: Be nice to your teammates! Tutor: I’m happy to work with our team :-) Tutor: m-hmm (showing attention) Adapted from Bales’ IPA (Bales, 195)

Experimental Design Hypotheses Students will be more engaged with agents displaying social behaviors Students will be sensitive to tutor goal orientation Interaction effect Frequent Green Infrequent Green Frequent Neutral None Power None Neutral Frequent Power Infrequent Neutral Infrequent Power None Green Social Behavior –Frequent, Infrequent, None Goal Alignment –Green, Power, Neutral 35 > Experimental Design >Manipulations

36 Measuring Student Bias Using a topic modeling tool – ccLDA [Paul and Girju, 2009] 36 > Experimental Design >Displaying Bias Corpus Collection1Collection2 Topic1 Topic 2 Topic 3

Example of Extracted Topics Heat quality right max decrease possible goes efficiency need gas graph say natural want goal fuel Tmax min sounds temp going friendly turbine kpa mean values different makes larger graphs bit green large kind produce hate steam team step solid 6574 split bored nat geo instead happens plant love yah blades sir dunno kk x85 rejected guessing starts FINAL life helping compromise nd depends corresponding teammate stays tmin new hard sitting afk tmax500 bec power decreases nuclear make 85 cycle work guess high pmin want pmax wait 570 lower green 40 tmax Pmin value low best point pick environment low sort 1 tutors effeciency 440 coool ecofriendly half fun 105 Nuclear sweeet maximized cooler question boy 6000 worked creepy Goes maxes generates makes 085 different qdot becuase decreasing click leads liquid gues doubt meet POWER 6574 DESIGN transfer hope Qin discussion km TOPIC 1 TOPIC 2 Background Green Power 37 > Experimental Design >Displaying Bias

Bias Measurement Metrics Max Topic-word bias: count the number of words in the list of the N most strongly associated words, and take the maximum across topics Average Topic-word bias: count the number of words in the list of the N most strongly associated words, and take the average across topics Weighted Topic-Word Average bias: Same but weight each word by its association within the background model first All three measures highly correlated both for Green and for Power perspectives Students in the Green condition got higher Green scores on average than Power scores and vice versa in the Power condition Only statistically significant for the first two metrics

Measuring Influence Within pairs, the Green score of the Green student and the Green score of the Power student were significantly correlated Same story for Power scores Result consistent with analysis of Politics dataset 39 > Experimental Design >Displaying Bias

Operationalization of Authoritativeness Negotiation coding scheme (Martin & Rose, 2007, Chapter 7) Agreement on K1/K2/Other.72 Kappa Coded all transcripts from Infrequent Social condition Authoritativeness score = K1/[K1 + K2] Within pairs, one Authoritative student and one NonAuthoritative student

Balance Effect Alignment Align: Authoritative partner shares affiliation with agent Neutral: Agent is neutral NoAlign: Non-Authoritative partner shares affiliation with agent Affiliated agent has polarizing effect on displayed bias Difference in bias scores was significantly higher in conditions with affiliated agents Direction of polarization depends on alignment Balance Effect Authoritative student shows less of his own bias when he’s in the minority NonAuthoritative student is less non- authoritative when he’s in the minority

Overview of Findings Topic models can display differences in goal orientation in chat data Confirmation of influence of partner speech on displayed bias Complex relationship between personal orientation, authoritativeness, and ingroup/outgroup effects

Outline Motivation from Opinion Mining Theoretical framework from Rhetoric and Discourse Analysis Study one: Political bias in a political discussion forum Study two: Goal orientation in chat based design discussions Current Directions

Full circle for opinion mining Continuing to operationalize multiple dimensions of relational codes Howley, I., Mayfield, E., & Rosé (to appear). Linguistic Analysis Methods for Studying Small Groups, in Hmelo-Silver, O’Donnel, & Chan (Eds.) International Handbook of Collaborative Learning, Taylor & Francis, Inc. Collaboration with Bob Kraut: investigating how exchange of social support is reflected in relational codes Collaboration with Bhiksha Raj: investigating evidence of relational codes like Negotiation in speech CHI submission in progress: Analysis of effects of bullying behavior on distribution of relational codes and learning New grant!: studying the emergence of leadership in ad-hoc teams

Computational Models of Discourse Analysis Focus on literature from the field of Discourse Analysis Investigating issues such as Conversational Structure, Attitude, Perspective, Persuasion and Positioning Critical reflection on the state-of-the art in language technologies Hands-on programming assignments, fun contests

Carolyn Penstein Rosé Gates-Hillman Center 5415