NAACL HLT 2010 | Los Angeles Thursday, June 3, 2010 Engaging Learning Groups using Social Interaction Strategies Rohit Kumar, Carolyn P. Rosé Language Technologies Institute Carnegie Mellon University

NAACL HLT 2010 | Los Angeles 2 Background Conversational Agents –Interactively supports Users One user in each interactive session –Variety of Tasks Information Access Education Therapy Entertainment

NAACL HLT 2010 | Los Angeles 3 Signal in the Noise < Outline CAs in Multi-Party Interactive Situations Social Interaction Strategies –Design –Implementation < Technical Basilica Architecture –Evaluation < Scientific Design Metrics Results

NAACL HLT 2010 | Los Angeles 4 Motivation Towards Conversational Agents for Multi-Party Interactive Situations –Challenges Technical: Agent Implementation Scientific: Agent Behavior / Interaction Design Group Interaction –Agents can help Kumar et. al., 2007 –Groups ignore/abuse Agents –Lack Social Communication Skills

NAACL HLT 2010 | Los Angeles 5 Small Group Communication Two Fundamental processes operate in group interaction Bales, 1950 Instrumental >Task-related vs. Expressive> Social-emotional Need for an Equilibrium Interaction Process Analysis (IPA) –Unit of Analysis: Utterance / Turn –12 Interaction Categories Shows Antagonism Shows Tension Disagrees Asks for Suggestion Asks for Opinion Asks for Orientation Gives Orientation Gives Opinion Gives Suggestion Agrees Shows Tension Release Shows Solidarity Instrumental Expressive +ve -ve

NAACL HLT 2010 | Los Angeles 6 Social Interaction Strategies Eleven Social Interaction Strategies developed –Based on Three Positive Social-Emotional Interaction Categories 1. Showing Solidarity: Raises other's status, gives help, reward 1a. Do Introductions: Introduce and ask names of all participants 1b. Be Protective & Nurturing: Discourage teasing 1c. Give Reassurance: When student is discontent, asking for help 1d. Complement / Praise: To acknowledge student contributions 1e. Encourage: When group or members are inactive 1f. Conclude Socially 2. Showing Tension Release: Jokes, laughs, shows satisfaction 2a. Expression of feeling better: After periods of tension, work pressure 2b. Be cheerful 2c. Express enthusiasm, elation, satisfaction: On completing task steps 3. Agreeing: Shows passive acceptance, understands, concurs, complies 3a. Show attention: To student ideas as encouragement 3b. Show comprehension / approval: To student opinions and orientations

NAACL HLT 2010 | Los Angeles 7 Implementation: Using Basilica Event-driven Architecture Kumar & Rosé, NAACL 2009 Rich-representational Power –Network of Behavioral Components Components are programmable –Using High-level Languages (Java) Flexibility to address Complex Interaction Dynamics –Four Agents developed using this Architecture Multi-Party Turn Taking Variety of Agent roles (Tutor, Peer, Mediator, …) Users with different roles in the same interactive session Reusability –Components –Agents in multiple Environments

NAACL HLT 2010 | Los Angeles 8 Implementation: WrenchTalk Tutor RequestDetector T.TakingCoordinator TutoringManager OutputCoordinator ConcertChatActorConcertChatListener ConcertChat Server TutoringActor PromptingActor MessageFilter DiscourseMemory PresenceFilter PromptingManager IntroductionsActor IntroductionsManager ActivityDetector AnnotationFilter PlanExecutor SocialController ProgressDetector Plug-In existing Components –TutoringManager / TutoringActor > Integrating TuTalk –IntroductionsManager/IntroductionsActor > State-based DM –PlanExecutor > Plan based DM –NLU > Classifiers / Annotators

NAACL HLT 2010 | Los Angeles 9 Implementation: WrenchTalk Tutor RequestDetector T.TakingCoordinator TutoringManager OutputCoordinator ConcertChatActorConcertChatListener ConcertChat Server TutoringActor PromptingActor MessageFilter DiscourseMemory PresenceFilter PromptingManager IntroductionsActor IntroductionsManager ActivityDetector AnnotationFilter PlanExecutor SocialController ProgressDetector Two primary controllers –PlanExecutor > Executes Task-related steps –SocialController > Triggers Social Behavior

NAACL HLT 2010 | Los Angeles 10 Implementation: Social Controller Social Behavior Triggering –Hand Crafted Rules Features –Last executed plan step –Annotations of student turns Dictionary Lookup –Activity Levels Groups & Individual Strategy: 1e. (Encourage) –Social Ratio Ratio of Social Turns to Task-related turns Threshold: 20% TutorOne last thing on this topic, Does more (or less) stress in a wrench make it easier to use? S95no change? TutorYou are correct about that.. Stress doesn't determine ease of use. TutorIt's the moment achieved by the wrench that determines the ease of use. S89yay! Tutor It's good to have your contributions in the discussion Jackie :-) TutorGo team :-) S89Go team yay TutorI am happy to be working with our team S89Me too S95whoa the bot knows my name Tutor Unfortunately maximum stress (12800) in our design1 is way above the maximum allowed stress (i.e. 8750) TutorThis wrench cannot be safely used! 1d. 2b.

NAACL HLT 2010 | Los Angeles 11 Research Context: Wrench Lab Freshmen Mechanical Engineering Course –Underlying concepts Force, Moment, Stress, Yield Strength, … Three part lab –Paperless Engineering CAD  CAA  CAM CAA includes Collaborative Wrench Design Activity –Teams of 3 – 4 students design a Wrench Fall 2009 Wrench Lab –98 Students

NAACL HLT 2010 | Los Angeles 12 Experiment: Evaluating Effectiveness Effectiveness of Social Behavior Vs. –Gold Standard> Human performance –Baseline > No Social Behavior Experimental Design –Three Conditions –Between Subjects –Groups & Tutors interact using ConcertChat TaskBaseline Same Instructional (Task-related) Behavior (37 steps) No Social Behavior SocialExperimentalAutomated Social Strategies HumanGold StandardHuman augmented Social Behavior

NAACL HLT 2010 | Los Angeles 13 Experiment: Survey 7 point Likert-scale Items Burke, 1967 Ratings about the Tutor Q1 I liked the tutor very much. Q2 The tutor was very cordial and friendly during the discussion Q3 The tutor was providing very good ideas for the discussion Q4 The tutor kept the discussion at a very comfortable level socially Q5 The tutor was part of my team Q6 The tutor received the ideas and suggestions I contributed to the discussion positively Ratings about the Activity Q7 I am happy with the discussion we had during the design challenge Q8 My group was successful at meeting the goals of the design challenge Q9 The design challenge was exciting and I did my best to come up with good designs Strongly Disagree Mostly Disagree Somewhat Disagree Neutral Somewhat Agree Mostly Agree Strongly Agree

NAACL HLT 2010 | Los Angeles 14 Experiment: Results Ratings About Tutor Agents with Social Behavior (Both Social & Human) rated better than No Social Behavior Automated Social Tutors significantly friendlier than No Social Behavior (Q2) Ratings About Activity> No Significant Difference w.r.t Baseline

NAACL HLT 2010 | Los Angeles Experiment: Analysis & Conclusion Significant benefits of employing Social Interaction Strategies –On Perceptual Metrics –Also, on Performance Metrics Current Implementation of Social Tutors not as good as Human tutors –Why? Possibly… Right Amount of Social Behavior Human tutors displayed significantly more Social Behavior Human = turnsSocial = turns Better Triggering Model implicit in Human Tutor Recent ITS 2010 Publication / Slides >