1. LTI Thesis Proposal Conversational Agents in Multi-Party Interactive Situations Rohit Kumar Committee Carolyn P. Rosé (Chair) Alan W. Black Ian R. Lane Jason D. Williams (AT&T Research) Friday, June 11, 2010

2. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 2 Bridges Collaborative Learning Communication Studies Small Group Communication Multi-Party Interaction Tutorial Dialog Social / Trust Games Dialog Systems Software Architecture Conversational Agents CSCW/CMC My Thesis

3. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 3 Conversational Agents (CAs) General Definition Conversational Agents are automated agents that extend conversation as a medium of interaction with machines. Many Applications Conversational agent is a system that exchanges information between user and agent using natural language dialogue - Sofus Macskassy... Conversational Agent, that is, an agent that can participate in fully natural dialog... - James Allen Conversational Agents exploit natural-language technologies to engage users in text-based information-seeking and task-oriented dialogs for a broad range of applications. - James Lester Conversational agents are communication technologies that use natural language and computational linguistic techniques to engage users in human-like, Web-based dialogs. - Wikipedia > Background > Conversational Agents

4. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 4 Some Conversational Agents Many studies have shown effectiveness of CAs – Information Access> Raux et. al., 2005 – Intelligent Tutoring> Kumar et. al. 2006/2007a – Therapy > Bickmore et. al., 2005 > Background > Conversational Agents

5. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 5 Bridges Collaborative Learning Communication Studies Small Group Communication Multi-Party Interaction Tutorial Dialog Social / Trust Games Dialog Systems Software Architecture Conversational Agents CSCW/CMC My Thesis

6. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 6 Multi-Party Interactive Situations (MPIS) Multi-Party Interactive Situations – Meetings, Dinner, Games, Classrooms – Groups more effective than Individuals at Intellective Tasks Review in McGrath, 1984 Increasing Computer Mediated Multi-Party Interaction – Instant Messaging / Chat – Video Conferencing – Collaborative Learning Beneficial pedagogical approach – Multi-player Online Games – Collaborative Work – Online Auctions – Social Networking > Background > Multi-Party Interactive Situations

7. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 7 Bridges Collaborative Learning Communication Studies Small Group Communication Multi-Party Interaction Tutorial Dialog Social / Trust Games Dialog Systems Software Architecture Conversational Agents CSCW/CMC My Thesis

8. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 8 CAs in Multi-Party Interactive Situations > Background > Conversational Agents in Multi-Party Interactive Situations One Agent + Two or More Users – E.g.: Tutor supporting Collaborative Learning Moderator in Group Interaction Two or More Agents + One User – E.g. TeamTalk Harris & Rudnicky, 2007 Two or More Agents + Two or More Users – E.g.: Multi-player Games (with multiple NPCs)

9. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 9 One Agent + Two or More Users – E.g.:Tutor supporting Collaborative Learning Moderator in Group Interaction CAs in Multi-Party Interactive Situations > Background > Conversational Agents in Multi-Party Interactive Situations CoBot Isbell et. al., 2000 Elva Tour Guide Zheng et. al., 2005 Multi-party Interaction Patterns Liu & Chee, 2004 Collaborative Learning - CycleTalk Kumar et. al., 2007a, 2007b Chaudhuri et. al., 2008, 2009 Situated Interaction Bohus & Horvitz, 2009 Stimulate Human Conversation Dohsaka et. al., 2009 Existing Work

10. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 10 CAs in MPIS: Two Challenges Building / Implementing Agents for Multi-Party Interactive Situations –Technical Challenge Engineering / Development Issues Designing Suitable / Useful Agent Behavior for such Situations –Scientific Challenge –Design Principles / Guidelines > Problem Statement

11. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 11 Technical Challenge: Building CAs > Problem Statement > Building CAs Existing approaches –Architectures Backboard, Pipeline, Hub, Multi-Expert, … –Representations State, Frame, Schema, Agenda, Plan, … Multi-Expert Architectures –Jaspis Turunen & Hakulinen, 2003 –RIME Nakano et. al., 2008 Event-driven Dialog Management –Olympus 2.0 > Interaction Manager Raux & Eskenazi, 2007

12. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 12 Technical Challenge: Building CAs Shortcomings of Existing approaches –Poor Representational Capability –Inflexibility to handle Complex Interaction Dynamics > Problem Statement > Building CAs

13. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 13 Technical Challenge: Building CAs > Problem Statement > Building CAs Poor Representational Capability –E.g. Scheduling policy for simultaneous responses from multiple users –Very High Level Languages Restricted to small set of operators and control structures Inability to freely combine representations in existing architectures

14. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 14 Technical Challenge: Building CAs > Problem Statement > Building CAs Inflexibility to handle Complex Interaction Dynamics –E.g. Ensuring all users get an equal opportunity to contribute Two Assumptions that Fail –Even Participation Assumption System Single User Interaction Multi-Party Interaction

15. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 15 Technical Challenge: Building CAs > Problem Statement > Building CAs Inflexibility to handle Complex Interaction Dynamics –E.g. Ensuring all users get an equal opportunity to contribute Two Assumptions that Fail –Known Addressee Assumption –Schisming Student1 OK, lets start Tutor What would happen to the power output of a Rankine Cycle at a higher operating temperature? Student2 hmmm … Can you answer that? Student1 I think it will increase. Tutor The correct answer is that at a higher operating temperature, more heat is added to the cycle and hence the power output increases too. What about the heat rejected by the cycle though? Tutor You are right S1. It increases too. Lets move on to the next topic. ? ?

16. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 16 Technical Challenge: Contributions > Problem Statement > Building CAs > Contributions Basilica A novel software architecture (and development tools) for building CAs in MPIS that provides –Rich Representational Capability Using High-Level Language –Flexibility to address Complex Interaction Dynamics Does not make Even Participation and Known Addressee assumptions –An array of CAs built using this architecture that provide Re-useable components

17. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 17 Basilica: Software Architecture Event-driven Architecture Agent Representation Kumar & Rosé, 2009 –Network of Behavioral Components –Components are programmable using High-level Languages Analogy: UI Toolkits –E.g.: Java Swing > Building CAs > Basilica MessageFilter TouchFilter TurnTakingCoordinator TutoringManager OutputCoordinatorSLActorSLListener Second Life Middleware Second Life Middleware TutoringActor GreetingActor

18. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 18 Basilica: Software Architecture Core Classes: Provide implementations for –Agent & Component Management –Event Propagation –Observer Interfaces –Agent Factory: Runtime Agent Assembly (from XML) –Utilities & Development Tools Re-Usable Components > Building CAs > Basilica

19. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 19 Basilica: Four Agents Four Agents built using Basilica –CycleTalk Tutor Agent –PsychChallenge Peer Agent –WrenchTalk Tutor Agent –Emergency Response (911) Interpreter Agent Four different Multi-Party Interactive Situations > Building CAs > Basilica > Agents Developed

20. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 20 CycleTalk Tutor Agent > Building CAs > Basilica > Agents Developed

21. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 21 CycleTalk Tutor Agent Implements Turn-taking rules suitable for Tutoring situation –Wait for responses from students –Prioritize between correct/incorrect/irrelevant responses > Building CAs > Basilica > Agents Developed MessageFilter RequestDetector TurnTakingCoordinator TutoringManager OutputCoordinatorConcertChatActorConcertChatListener ConcertChat Server TutoringActor PromptingActor HintingManagerHintingActor AttentionGrabbingFilterAttentionGrabbingActor

22. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 22 On Learning Portal of major Publishing Company PsychChallenge Peer Agent > Building CAs > Basilica > Agents Developed

23. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 23 PsychChallenge Peer Agent > Building CAs > Basilica > Agents Developed RoleFilter StatusFilter HintingActor GuessingActor OutputCoordinator M.WareActor M.WareListener Generic HTTP Middleware (Servlet) Generic HTTP Middleware (Servlet) PsychChallenge Web Interface (API) PsychChallenge Web Interface (API) HTTP RoleActor GreetingActor ScoreMemory WordMemory Agent has the same role as users

24. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 24 WrenchTalk Tutor Agent > Building CAs > Basilica > Agents Developed

25. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 25 WrenchTalk Tutor Agent > Building CAs > Basilica > Agents Developed RequestDetector T.TakingCoordinator TutoringManager OutputCoordinator ConcertChatActorConcertChatListener ConcertChat Server TutoringActor PromptingActor MessageFilter DiscourseMemory PresenceFilter PromptingManager IntroductionsActor IntroductionsManager ActivityDetector AnnotationFilter PlanExecutor SocialController ProgressDetector

26. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 26 NineOneOne Interpreter Agent > Building CAs > Basilica > Agents Developed VIDEOVIDEO

27. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 27 NineOneOne Interpreter Agent > Building CAs > Basilica > Agents Developed XMPPActor XMPP Connector PresenceFilter CallerTurnMemory NLUFilter RequestTypeFilter Parser ActClassifier MetaActManager IQAActManager SlotConfirmationManager MetaActor IQAActor SlotConfirmationActor TellCallerActor RequestDetector ReportDetector GenericsDetector SlotValueDetector InformDispatcherActor XMPPListener MessageFilter XMPP Server (Communication Backbone) XMPP Server (Communication Backbone) Distressed Caller Caller Proxy Live Audio ASR TTS Emergency Dispatcher

28. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 28 CAs in MPIS: Two Challenges Building / Implementing Agents for Multi-Party Interactive Situations –Technical Challenge Engineering / Development Issues Designing Suitable / Useful Agent Behavior for such Situations –Scientific Challenge –Design Principles / Guidelines > Problem Statement

29. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 29 Scientific Challenge: Agent Behavior > Problem Statement > Agent Behavior Unlike individual learners, Teams of students ignore / abuse automated tutors Kumar et. al. 2007a Agents lack Social Communication Skills –Need to be better communicators B9hi tutor B9tutor, you're a jerk B18I wonder if we can mute the tutor... B8maybe it stopped... Tutortype: HELP WITH CHANGETREHEAT if you want to discuss it with me. B8Okay B8guess i was wrong about that whole tutor stopping thing. B18Yeah, heh Tutor There will be more potential for power generation... Where might the increased heat input go instead? St16this tutor is really annoying St5Agreed a7so, got any ideas? TutorStarting now, you have around 40 minutes to work together… A2hrmmm a7stupid tutor no one asked you for help A20wow that tutor is annoying A13Yeah A13STFU tutor!

30. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 30 Bridges Collaborative Learning Communication Studies Small Group Communication Multi-Party Interaction Tutorial Dialog Social / Trust Games Dialog Systems Software Architecture Conversational Agents CSCW/CMC My Thesis Interaction & Performance Phase Movement Conflict Resolution

31. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 31 Scientific Challenge: Agent Behavior > Problem Statement > Agent Behavior > Small Group Communication Shows Solidarity Shows Tension Release Agrees Gives Suggestion Gives Opinion Gives Orientation Asks for Orientation Asks for Opinion Asks for Suggestion Disagrees Shows Tension Shows Antagonism Small Group Communication –Two Fundamental Processes –Bales, 1950 (Problem Solving Groups) Instrumental (task-related) vs. Expressive (social-emotional) Need for an Equilibrium Developed Interaction Process Analysis (IPA) –Bion, 1961 & Thelen, 1956 (Therapy Groups) Expressive Instrumental +ve -ve

32. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 32 Scientific Challenge: Agent Behavior CAs must perform both Task-related as well as Social Interaction Task-related Interaction strategies –Our earlier work Chaudhuri et. al., 2008, 2009 Social Behavior > Problem Statement > Agent Behavior

33. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 33 Related Work: Verbal Social Behavior by CAs Affective Computing Non-Verbal Social Behavior by ECAs Scientific Challenge: Social Behavior Emotional Intelligence in SDS André et. al., 2004 Relational Agents Bickmore & Picard, 2004 Eliciting Student Preferences Kumar et. al., 2007b Use of Emphatic Expressions Brave et. al., 2005 Dohsaka et. al., 2009 Polite Feedback Wang & Johnson, 2008 Humoroids Dybala et. al., 2009 Use of Backstories Bickmore et. al., 2009 > Problem Statement > Agent Behavior > Related Work Multi-Party

34. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 34 Scientific Challenge: Contributions > Problem Statement > Agent Behavior > Contributions Development of Social Interaction Strategies motivated from research in Small Group Communication Investigation of effectiveness of Social Interaction Strategies in MPIS –Experiments on amount, timing and type of behavior –Studied in multiple task domains Collaborative Learning & Trust Games Methodology & Materials to further the development of CAs as Good Communicators –Towards incorporating research from Small Group Communication in CA design

35. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 35 Social Interaction Strategies Eleven Social Interaction Strategies developed –Based on Three Positive Social-Emotional Interaction Categories > Social Behavior > Design 1. Showing Solidarity: Raises other's status, gives help, reward 1a. Do Introductions: Introduce and ask names of all participants 1b. Be Protective & Nurturing: Discourage teasing 1c. Give Reassurance: When student is discontent, asking for help 1d. Complement / Praise: To acknowledge student contributions 1e. Encourage: When group or members are inactive 1f. Conclude Socially 2. Showing Tension Release: Jokes, laughs, shows satisfaction 2a. Expression of feeling better: After periods of tension, work pressure 2b. Be cheerful 2c. Express enthusiasm, elation, satisfaction: On completing significant task steps 3. Agreeing: Shows passive acceptance, understands, concurs, complies 3a. Show attention: To student ideas as encouragement 3b. Show comprehension / approval: To student opinions and orientations

36. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 36 Implementation: WrenchTalk Tutor Agent RequestDetector T.TakingCoordinator TutoringManager OutputCoordinator ConcertChatActorConcertChatListener ConcertChat Server TutoringActor PromptingActor MessageFilter DiscourseMemory PresenceFilter PromptingManager IntroductionsActor IntroductionsManager ActivityDetector AnnotationFilter PlanExecutor SocialController ProgressDetector Two primary controllers –PlanExecutor > Executes Task-related steps –SocialController > Triggers Social Behavior Controllers regulate each other > By Blocking > Social Behavior > Implementation

37. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 37 Implementation: Social Controller Social Behavior Triggering –Hand Crafted Rules Four Features –Last executed plan step –Annotations of student turns Dictionary Lookup –Activity Levels Groups & Individual Strategy: 1e. (Encourage) –Social Ratio Ratio of Social Turns to Task-related turns Threshold: 20% TutorOne last thing on this topic, Does more (or less) stress in a wrench make it easier to use? S95no change? TutorYou are correct about that.. Stress doesn't determine ease of use. TutorIt's the moment achieved by the wrench that determines the ease of use. S89yay! Tutor It's good to have your contributions in the discussion Jackie :-) TutorGo team :-) S89Go team yay TutorI am happy to be working with our team S89Me too S95whoa the bot knows my name Tutor Unfortunately maximum stress (12800) in our design1 is way above the maximum allowed stress (i.e. 8750) TutorThis wrench cannot be safely used! 1d. 2b. > Social Behavior > Implementation

38. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 38 > Social Behavior > Experiments

39. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 39 Research Questions: Experiments Effectiveness of Social Behavior Vs. –Gold Standard (Human performance) –Baseline (No Social Behavior) Right Amount of Social Behavior Contribution of individual behaviors Effect of the Triggering Model Generalizability to other Interactive Situations Experiment 1 Proposed Task 1 & 2 Experiment 2 Experiment 3 Proposed Task 3 & 4 > Social Behavior > Experiments

40. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 40 Experiment 1-3: Interactive Situations Collaborative Design Labs –Mechanical Engineering Courses Freshmen: Wrench Design –Teams of 3-4 students –Underlying Concepts Force, Moment, Stress, Strength, … Sophomore: Power plant design –Teams of 2 students –Underlying Concepts Relationships between 4 System Properties & 5 Response Variables Tasks structured using Worksheets > Social Behavior > Experiments > Situations

41. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 41 Experiment 1-3: Methodology Controlled Experiments –Between subjects –Conducted over multiple sessions 35-40 minutes per session Different students in each session –Students randomly assigned to teams on the spot Team mates not seated next to each other Communicate using ConcertChat –Teams randomly assigned to conditions Nearly even distribution of conditions in each session Incentive –Class Credit –Gift Certificates for best team designs > Social Behavior > Experiments > Methodology

42. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 42 Experiment 1-3: Labs > Social Behavior > Experiments > Metrics

43. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 43 Experiment 1-3: Metrics Performance: Learning Outcomes –Pre & Post Tests –Multiple choice questions (MCQs) –Short Essay Questions (SEQs) Perception: Survey Burke, 1967 –7-point Likert-scale (1-Strongly disagree, 7-Strongly Agree) –9 items Ratings about Tutor (Agent): Likeable, Friendly, … Satisfaction Ratings: Task & Group Discussion Legitimacy > Social Behavior > Experiments > Metrics

44. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 44 Research Questions: Experiments Effectiveness of Social Behavior Vs. –Gold Standard (Human performance) –Baseline (No Social Behavior) Right Amount of Social Behavior Contribution of individual behaviors Effect of the Triggering Model Generalizability to other Interactive Situations Experiment 1 > Social Behavior > Experiments

45. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 45 Experiment 1: Evaluating Effectiveness Experimental Design –Three Conditions Benefit of the Collaborative Design Activity –No significant difference between pre-tests for the three conditions –Significant improvement from pre-test for post-test for all groups F(1,190) = 16.67, p <0.001, Effect size = 0.51 σ TaskBaseline Same Instructional (Task-related) Behavior No Social Behavior SocialExperimentalAutomated Social Interaction Strategies HumanGold StandardHuman augmented Social Behavior > Social Behavior > Experiment 1

46. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 46 Experiment 1: Results > Performance Learning Outcomes ANCOVA post-test = Fn(1, pre-test, condition, day) Significant effect of Condition F(2, 93) = 10.56, p < 0.001 Post-hoc Analysis –Task vs. Social { p < 0.01, 0.71 σ } –Task vs. Human { p < 0.001, 0.93 σ } –Social vs. Human { p = 0.509 } Tests: 8 MCQs & 3 SEQs Analysis done on total score > Social Behavior > Experiment 1 > Results

47. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 47 Experiment 1: Results > Perception Human & Social conditions better than Task conditions Human condition significantly better on (Q1 - Q5, Q8) Social condition significantly beter on Q2, marginal on Q4, Q5 > Social Behavior > Experiment 1 > Results

48. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 48 Experiment 1: Summary Significant benefits of employing Social Interaction Strategies Implementation of Social Tutors not as good as Human tutors –Right Amount of Social Behavior Significantly more Social Behavior in Human condition Average (Human) = 22.17Average (Social) = 16.83 –Human Social Behavior Triggering Experiment 2 Proposed Task 1 & 2 > Social Behavior > Experiment 1

49. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 49 Experiment 2: Amount of Behavior Manipulated by changing Social Ratio Experimental Design –Three Conditions Benefit of the Collaborative Design Activity –No significant difference between conditions on pre-tests –Significant improvements from pre- to post-test Effect size = 0.79 σ (Total), p < 0.001 None Same Instructional (Task-related) Behavior No Social Behavior (0%) LowSocial Ratio = 15% HighSocial Ratio = 30% > Social Behavior > Experiment 2

50. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 50 Experiment 2: Results Performance > Learning Outcomes Tests: 22 MCQs & 6 SEQs ANCOVA post-test = Fn(1, pre-test, condition, session) Significant effect of Condition on MCQs F(2, 97) = 3.48, p < 0.05 Post-hoc Analysis (for MCQs) –None vs. Low{ p < 0.07, 0.69 σ } –Low vs. High{ p < 0.07, 0.55 σ } –None vs. High{ not significant } Perception > Survey No Significant Differences between conditions > Social Behavior > Experiment 2 > Results Use moderate amount of Social Behavior

51. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 51 Research Questions: Experiments Effectiveness of Social Behavior Vs. –Gold Standard (Human performance) –Baseline (No Social Behavior) Right Amount of Social Behavior Contribution of individual behaviors Effect of the Triggering Model Generalizability to other Interactive Situations Experiment 3 > Social Behavior > Experiments

52. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 52 Experiment 3: Types of Behavior Experimental Design –Six Conditions NothingNo Social Behavior0 AgreeingShow Attention & Comprehension2 FriendlyBe Protective & Nurturing Give Reassurance Complement/Praise Encourage 4 Intro + OutroIntroduce & Conclude Socially2 Agreeing + Friendly Show Attention & Comprehension Be Protective & Nurturing Give Reassurance Complement/Praise Encourage 6 EverythingAll Strategies from Experiment 111 > Social Behavior > Experiment 3

53. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 53 Experiment 3: Types of Behavior Benefit of the Collaborative Design Activity –No significant difference between Pre-Test for all conditions –Significant difference between Pre- & Post-Test F(1, 227) = 5.67, p < 0.05, Effect = 0.3 σ Tests: 8 MCQs & 3 SEQs Results –No Significant Differences between conditions for MCQs Survey Pending –Analysis of SEQs > Social Behavior > Experiment 3 > Results

54. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 54

55. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 55 Research Questions: Experiments Effectiveness of Social Behavior Vs. –Gold Standard (Human performance) –Baseline (No Social Behavior) Right Amount of Social Behavior Contribution of individual behaviors Effect of the Triggering Model Generalizability to other Interactive Situations Proposed Task 1 & 2 Proposed Task 3 & 4 > Social Behavior > Experiments

56. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 56 Tasks: Effect of the Triggering Model Modeling human social behavior triggering 1.Annotate available corpus of Human Triggering Behavior Dimensions IPA (Coarse-grained / Fine-grained) Adjacency relationships Crowd-sourced Annotation 2.Setup Feature Extractors - Partially Done 3.Modeling Annotated Data Supervised Learning Evaluation > Proposed Tasks > Task 1

57. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 57 Tasks: Effect of the Triggering Model Evaluating the effect of human-like triggering –Compare a.Rule-based Triggering Model b.Random Triggering Model c.Experimental Triggering Model –Additional Factor Amount of Social Behavior ( None, Low, High) –Opportunities: Fall10 1.Wrench Design Lab (CMU/Drexel) 2.Power Plant Design Lab Contingent on results from Task 1 > Proposed Tasks > Task 2

58. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 58 Tasks: Generalizability to other MPIS Interactive Situation > Real-Time Auction –Setup to induce Competition & Cooperation Task 3: Development –Infrastructure Development Chatroom + Auctioning Interface –Material Development Rules & Instructions –Agent Development Agent Role: Auction Participant Adapting Social Interaction Strategies > Proposed Tasks > Task 3

59. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 59 Tasks: Generalizability to other MPIS Evaluating generalizability of Social Interaction Strategies 1.Pilot Studies 2.Actual Studies Around 120 subjects (25-30 groups) Compare: Agent Behavior –Task Only –Task + Social Measures –Performance: Groups Task Success / Mean Expenditure –Perception: Agent, Task > Proposed Tasks > Task 4

60. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 60 Proposed Tasks: Summary 1.Modeling human social behavior triggering –Annotate, Setup, Model 2.Evaluating the effect of human-like triggering 3.Online Auctioning Situation: Development –Infrastructure, Materials, Agent 4.Evaluating generalizability of Social Interaction Strategies > Proposed Tasks

61. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 61 Tentative Timeline JunJulAugSepOctNovDecJanFebMarAprMayJunJulAug Milestones Exp. 3 Cont. Task 1 Task 2 Task 3 Task 4 10-701 Conferences Writing Job Search > Proposed Tasks > Timeline Defense Trigger Modeling & Validation Auction Situation Development & Experiment

62. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 62 Potential Contributions: Summary Basilica Software architecture for building CAs in MPIS that provides –Rich Representational Capability –Flexibility to address Complex Interaction Dynamics –An array of CAs built using this architecture that provide Re-useable components Development of Social Interaction Strategies motivated from research in Small Group Communication Investigation of effectiveness of Social Interaction Strategies in MPIS Methodology & Materials to further the development of CAs as Good Communicators > Potential Contributions > Summary Questions

63. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations Publications (Kumar et. al., 2007a) Rohit Kumar, Carolyn Rosé, Mahesh Joshi, Yi-Chia Wang, Yue Cui and Allen Robinson, 2007, Tutorial Dialogue as Adaptive Collaborative Learning Support, 13th AIED 2007, Los Angeles, California (Kumar et. al., 2007b) Rohit Kumar, Gahgene Gweon, Mahesh Joshi, Yue Cui, Carolyn Rosé, Supporting students working together on Math with Social Dialogue, Workshop on Speech and Language Technology in Education, Farmington, PA, 2007 (Chaudhuri et. al., 2008) Sourish Chaudhuri, Rohit Kumar, Carolyn P. Rosé, 2008, Its not easy being green - Supporting Collaborative Green Design Learning, ITS 2008, Montreal (Chaudhuri et. al., 2009) Sourish Chaudhuri, Rohit Kumar, Iris Howley, Carolyn P. Rosé, 2009, Engaging Collaborative Learners with Helping Agents, Proc. of AI in Education (Kumar and Rosé, 2009) Rohit Kumar, Carolyn Rosé, 2009, Building Conversational Agents with Basilica, Proc. of NAACL-HLT, Boulder, CO (Kumar and Rosé, 2010b) Rohit Kumar, Carolyn P. Rosé, 2010, Engaging learning groups using Social Interaction Strategies, NAACL- HLT, Los Angeles, CA (Kumar and Rosé, 2010a) Rohit Kumar, Carolyn P. Rosé, 2010, Conversational Tutors with Rich Interactive Behaviors that support Collaborative Learning, Workshop on Opportunities for intelligent and adaptive behavior in collaborative learning systems, ITS 2010, Pittsburgh, PA (Kumar et. al., 2010) Rohit Kumar, Hua Ai, Jack Beuth, Carolyn P. Rosé, 2010, Socially-capable Conversational Tutors can be Effective in Collaborative-Learning situations, Intelligent Tutoring Systems, Pittsburgh, PA (Ai et. al., 2010) Hua Ai, Rohit Kumar, Amrut Nagasunder, Carolyn P. Rosé, 2010, Exploring the Effectiveness of Social Capabilities and Goal Alignment in Computer Supported Collaborative Learning, Intelligent Tutoring Systems, Pittsburgh, PA 63

64. LTI Thesis ProposalConversational Agents in Multi-Party Interactive Situations 64 Less Aggressive Timeline JunJulAugSepOctNovDecJanFebMarAprMayJunJulAugSepOctNovDec Milestones Exp. 3 Cont. Task 1 Task 2 Task 3 Task 4 10-701 Conferences Writing Job Search > Proposed Tasks > Timeline Defense Trigger Modeling & Validation Auction Situation Development & Experiment