Carolyn Penstein Rosé Language Technologies Institute and Human-Computer Interaction Institute With funding from the National Science Foundation and the Office of Naval Research

Human-Computer Interaction LinguisticsTechnology ? Learning Sciences

LISTEN!LISTEN!

Human-Computer Interaction LinguisticsTechnology

Outline Overview Technology for Dynamic Collaborative Learning Support Study One: Agents Offering Student Control Study Two: Agents Behaving Socially Conclusions and Current Directions

Outline Overview Technology for Dynamic Collaborative Learning Support Study One: Agents Offering Student Control Study Two: Agents Behaving Socially Conclusions and Current Directions

Helping students learn together in on-line groups… Engineering Math Psychology Earth Sciences Higher Education Middle School High School Machine Learning

Collaborative Design Task Instructional Goal: Apply thermodynamics to design of a power plant based on the Rankine Cycle paradigm Each pair turns in exactly one design Competing Student Goals: Power: Design a power plant that achieves maximum power output Motivated by economic concerns Green: Design a power plant that has the minimum impact on the environment Motivated by environmental concerns

Dynamic Collaborative Learning Support Support for collaborative learning is like training wheels Effective support allows learners to achieve better collaboration Unnecessary support can be demotivating Fading support is ideal But too little support can be detrimental as well Human facilitators are able to achieve the right balance

Outline Overview Technology for Dynamic Collaborative Learning Support Study One: Agents Offering Student Control Study Two: Agents Behaving Socially Conclusions and Current Directions

Technology for Dynamic Collaborative Learning Support Identify conversational interactions that are valuable for learning Automatic conversation analysis Automates assessment of group processes Enables context sensitive triggering of support Interactive support technologies

12 Classroom Studies Chat room style interaction ConcertChat 2 nd year Undergraduate Mechanical Engineering students Classroom session During the semester Instruction  Pretest  Session  Posttest  Questionnaire 12 > Experimental Design >Study Procedure

Basilica Architecture (Kumar & Rosé, in press) Easy integration with a variety of computer supported collaborative learning environments RequestDetector T.TakingCoordinator TutoringManager OutputCoordinator ConcertChatActorConcertChatListener ConcertChat Server TutoringActor PromptingActor MessageFilter DiscourseMemory PresenceFilter PromptingManager IntroductionsActor IntroductionsManager ActivityDetector AnnotationFilter PlanExecutor SocialController ProgressDetecto r

Download tools at: Monitoring Collaboration with Machine Learning Technology TagHelper Labeled Texts Unlabeled Texts Labeled Texts A Model that can Label More Texts Time Behavior <Triggered Intervention>

Using Conversational Agents to Support Collaboration

Overview of Findings Dialogue agents more effective than hints (Rosé et al., 2001; Rosé et al., 2005; Kumar et al., 2006) Collaborative learning with the support of dialogue agent significantly better than working alone (Kumar et al., 2007) 1 S.D. = 1 letter grade Working with a partner and an agent is 1.24 S.D. better Working with a partner is 1 S.D. better Working alone with an agent is 1 S.D. better

Overview of Findings Students learn more when they have more control (Chaudhuri et al., 2009) Students learn more from agents that employ social strategies (Kumar et al., 2010; Ai et al., 2010) Students explore design space more completely when agents show more openness to alternatives ideas (Kumar et al., under review)

Outline Overview Technology for Dynamic Collaborative Learning Support Study One: Agents Offering Student Control Study Two: Agents Behaving Socially Conclusions and Current Directions

Student Control Student: so i think the best thing to go with first is to increase the pressure entering the connector Tutor: You might want to review what you saw earlier in the sensitivity analysis investigating the effect of changing P-max on steam quality and efficiency, from the top half of page 13 in your book. Tutor: type: HELP WITH CHANGEPMAX if you want to discuss it with me. Student: help with changepmax Tutor: Remember the sensitivity analysis you have completed where you manipulated P-Max and observed it's effect on the efficiency of the cycle. You can refer back to Figure 11 on page 13 of your booklet, which …

Student Control Hypotheses: Dialogue Agents increase conceptual depth Students will be more engaged when they are offered control over timing Pointer Hints Suggestion that students think about a certain issue, with a pointer to a page in their book. Dialogue Agents Interactive “walk through” a line of reasoning found on the corresponding page in the book No Support Dialogue Only Pointer Only Pointer+ Dialogue

Example Interaction

Experimental Procedure Consent form and Pre-Questionnaire (10 minutes) Training (15 minutes) Instruction on Rankine Cycles and Green Design (65 minutes) Pretest (15 minutes) Collaborative Design Exercise (40 minutes) Packaging up the design (10 minutes) Posttest and Post-Questionnaire (20 minutes)

Pointer Only Student: so i think the best thing to go with first is to increase the pressure entering the connector Tutor: You might want to review what you saw earlier in the sensitivity analysis investigating the effect of changing P-max on steam quality and efficiency, from the top half of page 13 in your book. …

Dialogue Only Student: so i think the best thing to go with first is to increase the pressure entering the connector Tutor: Remember the sensitivity analysis you have completed where you manipulated P-Max and observed it's effect on the efficiency of the cycle. You can refer back to Figure 11 on page 13 of your booklet, which I have pasted onto your whiteboard for your convenience. You can delete it after this conversation with me if it is in your way. Having seen how changing P-Max, the Maximum Pressure at the Heater Inlet, effects efficiency and Steam, do you think as a general principle the Maximum pressure should be increased ? Student: Yes Tutor: So the Efficiency of a Cycle increases by…

Pointer + Dialogue Student: so i think the best thing to go with first is to increase the pressure entering the connector Tutor: You might want to review what you saw earlier in the sensitivity analysis investigating the effect of changing P-max on steam quality and efficiency, from the top half of page 13 in your book. Tutor: type: HELP WITH CHANGEPMAX if you want to discuss it with me. Student: help with changepmax Tutor: Remember the sensitivity analysis you have completed where you manipulated P-Max and observed it's effect on the efficiency of the cycle. You can refer back to Figure 11 on page 13 of your booklet, which …

Results ANCOVA Model Dependent Variable: Posttest score Independent Variables: Dialogue and Pointer Covariate: Pretest score Significant effect of Pointer Hints F(1,71) = 7.5, p <.01, effect size.57 s. d. Dialogue with and without Pointer hints,.8 s. d. PretestPosttest Lab Session A (Pointer Hint Condition)17.9 (5.9)18.8 (2.9) Lab Session B (Pointer +Dialogue condition)14.8 (7.0)19.1 (2.9) Lab Session C (Dialogue condition)14.4 (5.7)15.7 (4.8) Lab Session D (No Support condition)17.1 (3.9)17.3 (4.0)

Results Only Pointer + Dialogue has significant pre to post test gain Only Pointer + Dialogue is significantly better than No Support PretestPosttest Lab Session A (Pointer Hint Condition)17.9 (5.9)18.8 (2.9) Lab Session B (Pointer +Dialogue condition)14.8 (7.0)19.1 (2.9) Lab Session C (Dialogue condition)14.4 (5.7)15.7 (4.8) Lab Session D (No Support condition)17.1 (3.9)17.3 (4.0)

What was the impact of the experimental manipulation on displayed reasoning? ANCOVA Model Dependent Variable: Number of Reasoning Expressions Independent Variables: Pointer and Dialogue Covariate: Total amount of talk in dialogue

What was the impact of the experimental manipulation on displayed reasoning? Significant effect of Dialogue on Amount of reasoning F(1,71) = 7.9, p <.01 Effect size.35 s.d.

Outline Overview Technology for Dynamic Collaborative Learning Support Study One: Agents Offering Student Control Study Two: Agents Behaving Socially Conclusions and Current Directions

Experimental Design Hypotheses Students will be more engaged with agents displaying social behaviors Students will be sensitive to tutor goal orientation Interaction effect Frequent Green Infrequent Green Frequent Neutral None Power None Neutral Frequent Power Infrequent Neutral Infrequent Power None Green Social Behavior –Frequent, Infrequent, None Goal Alignment –Green, Power, Neutral 31 > Experimental Design >Manipulations

An Example of Displaying Bias 32 > Experimental Design >Agent Design Green Bias Green: What is bad about increasing heat input to the cycle is that more waste heat is rejected to the environment. Neutral and Power: Increasing heat input to the cycle increases waste heat rejected to the environment. Power Bias: Power: What is good about increasing heat input to the cycle is that more power output is produced. Neutral and Green: Increasing heat input to the cycle increases power output produced.

33 Example of Social Behaviors 33 > Experimental Design >Agent Design 1. Showing Solidarity: Raises other's status, gives help, reward 2. Showing Tension Release: Jokes, laughs, shows satisfaction 3. Agreeing: Shows passive acceptance, understands, concurs, complies Tutor: Let’s Introduce ourselves. My name is Avis. Tutor: Be nice to your teammates! Tutor: I’m happy to work with our team :-) Tutor: m-hmm (showing attention) Adapted from Bales’ IPA (Bales, 195)

Experimental Design Hypotheses Students will be more engaged with agents displaying social behaviors Students will be sensitive to tutor goal orientation Interaction effect Frequent Green Infrequent Green Frequent Neutral None Power None Neutral Frequent Power Infrequent Neutral Infrequent Power None Green Social Behavior –Frequent, Infrequent, None Goal Alignment –Green, Power, Neutral 34 > Experimental Design >Manipulations

Results ANCOVA Model Dependent Variable: Post-test score Independent Variables: Social and Match Covariate: Pre-test score Significant effect of Social behavior F(2,94) = 5.27, p <.01 Infrequent is significantly better than the other two (effect size.83 s.d.) Marginal interaction between Social Behavior and Match Infrequent Social only better when Match is true Frequent Green Infrequent Green Frequent Neutral None Power None Neutral Frequent Power Infrequent Neutral Infrequent Power None Green Match: For Green students: Green = Match Power = NoMatch Neutral = Neutral For Power students: Green = NoMatch Power = Match Neutral = Neutral

Other Effects of Social Behavior For each student, computed number of contributions that were: Positive social, Off-task, Negative comments about tutor Students contribute more Positive social contributions in Social conditions Students contribute more Off-task turns in the NoSocial condition Students contribute more Negative comments about the tutor in the Frequent social conditions

37 Measuring Student Bias Using a topic modeling tool – ccLDA [Paul and Girju, 2009] 37 > Experimental Design >Displaying Bias Corpus Collection1Collection2 Topic1 Topic 2 Topic 3

Example of Extracted Topics Heat quality right max decrease possible goes efficiency need gas graph say natural want goal fuel Tmax min sounds temp going friendly turbine kpa mean values different makes larger graphs bit green large kind produce hate steam team step solid 6574 split bored nat geo instead happens plant love yah blades sir dunno kk x85 rejected guessing starts FINAL life helping compromise nd depends corresponding teammate stays tmin new hard sitting afk tmax500 bec power decreases nuclear make 85 cycle work guess high pmin want pmax wait 570 lower green 40 tmax Pmin value low best point pick environment low sort 1 tutors effeciency 440 coool ecofriendly half fun 105 Nuclear sweeet maximized cooler question boy 6000 worked creepy Goes maxes generates makes 085 different qdot becuase decreasing click leads liquid gues doubt meet POWER 6574 DESIGN transfer hope Qin discussion km TOPIC 1 TOPIC 2 Background Green Power 38 > Experimental Design >Displaying Bias

Bias Measurement Metrics Max Topic-word bias: count the number of words in the list of the N most strongly associated words, and take the maximum across topics Average Topic-word bias: count the number of words in the list of the N most strongly associated words, and take the average across topics Weighted Topic-Word Average bias: Same but weight each word by its association within the background model first All three measures highly correlated both for Green and for Power perspectives Demonstrates convergent validity Students in the Green condition got higher Green scores on average than Power scores and vice versa in the Power condition Demonstrates face validity of construct

Mapping Influence Per student, Green and Power scores were negatively correlated Demonstrates divergent validity Within pairs, Green scores of both students positively correlated, same with Power scores Suggests positive collaboration Students displayed more bias when the agent was biased Suggests that Agent bias enhances the effect of the student level goal manipulation 40 > Experimental Design >Displaying Bias

What did we learn about designing dialogue agents? Dialogue agent technology is effective for supporting collaborative learning Can be triggered through automatic text analysis Increases amount of explanation Increases learning Design of agent behavior must be sensitive to social concerns Students are more receptive when they have control over timing Students engage better with agents that are on their side Social behavior is effective when it is not too frequent

Outline Overview Technology for Dynamic Collaborative Learning Support Study One: Agents Offering Student Control Study Two: Agents Behaving Socially Conclusions and Current Directions

Current Directions Further exploration of bias and influence in collaborative discussions Tutor could always appear to be on the student’s side Complex Management of Attention Engagement between student and agent Engagement between student and partner New! Exploration insights from theories of Pragmatics and Sociolinguistics for detecting social dynamics from speech

The (Near!) Future of Online Group Work

Assessment from speech *Using prosodic features rather than content 45 Speech PredictionReport

46 LevelMarkednessShift State of the ArtPragmaticSociolinguistic Encoding Insights from Linguistics into Representation of Speech TIME STYLE VALUE

Social aspects that are directly connected to independent prosodic features Emotions e.g. annoyance, anger, sadness, boredom [Ang et al 02, Lee&Narayanan 02, Liscombe et al 03] High accuracy (70 ~ 90%) Social aspects that are not directly connected - Personality e.g. charisma, extroversion [Rosenberg & Hirschbert 05, Mairesse et al. 07] - Low correlation values reported (r=0.2 ~ 0.45) 47 Prior work on analysis of social meaning from speech

48 Definition: Process of taking up, transforming, or building on an idea expressed earlier in a conversation Example: Let’s use plastic since it’s so cheap. I understand that plastic is cheaper, but paper is better for the environment No, that won’t work ✔ ✖ What about idea Co-Construction?

92 prosodic features Random baseline - Recall: 0.17, precision: 0.15, f-score: 0.16, kappa: 0.04 Result - Recall: 0.72, precision: 0.24, f-score: 0.35, kappa: 0.23 What about idea Co-Construction? MFCC (power) 4 Pitch related 4 Amplitude related 4 Energy (intensity) related Segment duration Speaker change 3 phoneme related 48 Phonemes

LISTEN!LISTEN!

New Direction: Inspired by Linguistics! Previous work considers each feature independently using an absolute scale High energy => idea co-construction (ICC) Slow speech rate => ICC However, features are interdependent, and how they are grouped together represent different speech styles [Eckert & Rickford, 2001; Labov, 2001] {“Respect”} => ICC 51

Style and Feature Sets Styles are combinations of prosodic features that co-occur with one another Example: showing that one is listening respectfully I understa::nd that plastic is cheaper, {pause} but paper is better for the environment {said more quietly}. Subsets of the same features don’t mean what they mean as a set That’s the stu::pidest idea I have heard today! No::! {pause} That won’t work {said loudly}

Considering Style (Level) 53 Level TIME STYLE VALUE

Considering Markedness Expected speech style differs depending on the context of speech [Labov, 2001; Myers-Scotton; Baugh, 2001; Bell, 2001] Unusual (a.k.a Marked) styles carry a social interpretation [Grice, 1975; Brown & Levinson, 1987] If people speak in an unusual way within the given context, that triggers an inferential meaning Expected levels of showing respectful listening might vary depending upon the pre-existing relationship between speakers 54

Considering Markedness 55 LevelMarkedness State of the ArtPragmatic STYLE VALUE TIME

Considering Style Shift Previous work in sociolinguistics shows that style shift can tell us something about interpersonal relationships Speakers show respect by imitating their partners [Giles & Powesland, 1975; Labov, 1972, 1982, 1984; Baugh, 2001; Eckert, 2001] Speakers build solidarity by de-emphasizing differences and emphasizing similarities [Goffman, 1955] 56

57 LevelMarkednessShift State of the ArtPragmaticSociolinguistic Considering Style Shift TIME STYLE VALUE

58 LevelMarkednessShift State of the ArtPragmaticSociolinguistic Encoding Insights from Linguistics into Representation of Speech TIME STYLE VALUE

Carolyn Penstein Rosé Gates-Hillman Center 5415