1. Conversational Strategies that Support Idea Generation Productivity in Groups Rohit Kumar Jack L. Beuth Carolyn P. Rosé

2. Overview User Study: – Design of Tutor prompts for supporting Collaborative Learning Software Agents: Monitoring Student Interactions – Social Interaction Strategies – Presentation Style Outline: – Background: Research Objectives, Earlier Work – Presentation Style: Heteroglossia – User Study: Task, Procedure, Metrics, Design – Results 2 Background  Presentation Style  User Study  Idea Generation  Results

3. Bridges 3 Collaborative Learning Communication Studies Small Group Communication Tutorial Dialog Dialog Systems Conversational Tutors Our Work Background  Presentation Style  User Study  Idea Generation  Results

4. Prior Work Benefits of – 0.90 σ : Collaborative Learning – 1.06σ : Dynamic Support (Software Agents) – 1.24σ : Combined Kumar et. al., 2007 Students ignore/abuse Tutors – Tutors are unable to manage the attention of the students 4 A20wow that tutor is annoying A13Yeah A13STFU tutor! B9hi tutor B9tutor, you're a jerk a7so, got any ideas? TutorStarting now, you have around 40 minutes to work together… A2hrmmm a7stupid tutor no one asked you for help Tutor There will be more potential for power generation... Where might the increased heat input go instead? St16this tutor is really annoying St5Agreed Background  Presentation Style  User Study  Idea Generation  Results

5. Social Interaction Strategies Small Group Communication – Instrumental vs. Socio-Emotional Interaction Bales, 1950 5 Background  Presentation Style  User Study  Idea Generation  Results Shows Solidarity Shows Tension Release Agrees Gives Suggestion Gives Opinion Gives Orientation Asks for Orientation Asks for Opinion Asks for Suggestion Disagrees Shows Tension Shows Antagonism 1. Showing Solidarity: Raises other's status, gives help, reward 1a. Do Introductions: Introduce and ask names of all participants 1b. Be Protective & Nurturing: Discourage teasing 1c. Give Reassurance: When student is discontent, asking for help 1d. Compliment / Praise: To acknowledge student contributions 1e. Encourage: When group or members are inactive 1f. Conclude Socially 2. Showing Tension Release: Jokes, laughs, shows satisfaction 2a. Expression of feeling better: After periods of tension, work pressure 2b. Be cheerful 2c. Express enthusiasm, elation, satisfaction: On completing significant task steps 3. Agreeing: Shows passive acceptance, understands, concurs, complies 3a. Show attention: To student ideas as encouragement 3b. Show comprehension / approval: To student opinions and orientations Kumar & Rosé, 2010

6. Social Interaction Strategies Implemented Using Basilica Kumar & Rosé, 2011 (IEEE Trans. Learning Tech.) 6 Background  Presentation Style  User Study  Idea Generation  Results 139TutorOne last thing on this topic, Does more (or less) stress in a wrench make it easier to use? 140S95no change? 141TutorYou are correct about that.. Stress doesn't determine ease of use. 142TutorIt's the moment achieved by the wrench that determines the ease of use. 143S89yay! 144TutorIt's good to have your contributions in the discussion Jackie :-) 145TutorGo team :-) 146S89Go team yay 147TutorI am happy to be working with our team 148S89Me too 149S95whoa the bot knows my name 150TutorUnfortunately maximum stress (12800) in our design1 is way above the maximum allowed stress (i.e. 8750) 151TutorThis wrench cannot be safely used!

7. Social Interaction Strategies Significant Benefits – 0.71 σ : Using Fully-Automated Social Interaction Strategies Kumar et. al., 2010 (ITS) – More Efficient at delivering the instructional content Other Experiments – Amount of Social Behavior Learning, On-Task Interaction Hua et. al., 2010 (ITS) – Why/How & Timing of Behavior Learning, Perception of TutorKumar & Rosé, 2011(SigDial) – Transfer to other domains Current Work – Style of Presentation This talk 7 Background  Presentation Style  User Study  Idea Generation  Results

8. Presentation Style Heteroglossia vs. Monoglossia – Lexical/Grammatical choices – Indicates social positioning of the speaker Heteroglossia – Awareness/Acknowledgement of multiple perspectives 8 Background  Presentation Style  User Study  Idea Generation  Results Original/NeutralThink about how we can improve this? HeteroglossicI suggest that you think about how to improve this MonoglossicThink about how to improve this Original/NeutralMaybe we can use a safer material this time! HeteroglossicUsing a safer material would be strongly advisable! MonoglossicUse a safer material this time!

9. User Study Design – 2 x 3 – Full factorial – Between subjects Using Social Interaction Strategies – Yes (S) – No (NS) Style of Presentation For Tutor Prompts – Original (N) – Monoglossic (M) – Heteroglossic (H) 9 Background  Presentation Style  User Study  Idea Generation  Results Social Interaction Strategies Presentation Style N + NSN + S M + NSM + S H + NSH + S

10. User Study: Learning Activity Mechanical Engineering Course – Freshmen: Wrench Design Underlying Concepts – Force, Moment, Stress, Strength, … – Collaborative Design Lab Design a better wrench – Ease of Use, Cost, Safety Team of 3-4 students Communicate using ConcertChat Supported by Automated tutors – Randomly assigned to conditions 10 Background  Presentation Style  User Study  Idea Generation  Results

11. User Study: Procedures 1.Pre-Test: 10 minutes – Qs: 7 multiple-choice, 4 brief explanation 2.Collaborative Design Competition: 35 minutes – Interact with tutors 3.Post-Test: 10 minutes 4.Perception Survey: 5 minutes Details in the paper – 10 items 7-point Likert scale – Personality descriptors 11 Background  Presentation Style  User Study  Idea Generation  Results

12. User Study Metrics – Learning Outcomes – Perception Ratings (of Tutor, Task) – Idea Contributions (by Students) Count is estimated using dictionary – how about, can we, we should, … – increase, decrease, steel, … Noisy Measure – For all conditions 12 Background  Presentation Style  User Study  Idea Generation  Results

13. Idea Generation: Review Idea Generation & Learning Wang et. al., 2007 (Upcoming IJCSCL Article) Depends on the Task Idea Generation in Groups – Process Loss Affect Productivity  Reduce Learning Objective: – To create support that enable the benefits of idea generating interactions with negative side-effects Instrument: – Automatically generated intervention prompts 13 Background  Presentation Style  User Study  Idea Generation  Results

14. Results Learning Outcomes: – No significant effects of either factor Perception Ratings: – Heteroglossic prompts: Happier with the discussion p<0.05 As good as human tutorp=0.08 Perceived task successp<0.08 – Heteroglossic & Monoglossic prompts (i.e. Style Consistency) Liked the tutor very muchp<0.05 – With Social Interaction Strategies Tutor perceived more supportive & less pushyp<0.05 14 Background  Presentation Style  User Study  Idea Generation  Results

15. Results Idea Generation Productivity – As estimated by counts of idea contributions Social Interaction Strategies – No Significant Effect Presentation Style – Heteroglossic > Original – Heteroglossic > Monoglossic Interaction Effect p<0.05 – Heteroglossic prompts could compensate for lack of social capabilities 15 Background  Presentation Style  User Study  Idea Generation  Results

16. Conclusions Exploration of style / function of tutor prompts Among other factors (amount, timing, transfer, …) – Perception Ratings Preference towards Heteroglossic prompts & the use of Social behavior – Idea Generation Heteroglossic style encourages more productive idea generation – Observation: Unlike Wang et. al. 2007, no relationship between idea generation & learning 16

17. Thank You Acknowledgements – Colleagues Gregory Dyke – CMU Mechanical Engineering Faculty & TAs Maarten P. De Boer, Philip Leduc, Robert Reid, Joshua Eickmeier, Mallory Elbert, Lina Gonzalez, Prerna Singh – C-MITES Ann Shoplik, Pamela Piskurich – National Science Foundation Questions

18. 18

19. Prior Work Benefits of – 0.90 σ : Collaborative Learning – 1.06σ : Adaptive Support (Tutor Agents) – 1.24σ : Combined Kumar et. al., 2007 Students ignore/abuse Tutors Remedial Interaction Tactics – Attention Grabbing Chaudhuri et. al., 2008/2009 19 A20wow that tutor is annoying A13Yeah A13STFU tutor! B9hi tutor B9tutor, you're a jerk a7so, got any ideas? TutorStarting now, you have around 40 minutes to work together… A2hrmmm a7stupid tutor no one asked you for help Tutor There will be more potential for power generation... Where might the increased heat input go instead? St16this tutor is really annoying St5Agreed Background  Presentation Style  User Study  Idea Generation  Results

20. Presentation Style Heteroglossia vs. Monoglossia – Lexical/Grammatical choices – Indicates social positioning of the speaker Heteroglossia – Awareness/Acknowledgement of multiple perspectives – Criteria Martin & White, 2005 – Propositional content must be asserted – Awareness to alternative perspective must be visible – Awareness is expressed grammatically or paraphrastically 20 Background  Presentation Style  User Study  Idea Generation  Results

21. Presentation Style Examples 21 Background  Presentation Style  User Study  Idea Generation  Results Original/NeutralThink about how we can improve this? HeteroglossicI suggest that you think about how to improve this MonoglossicThink about how to improve this Original/NeutralMaybe we can use a safer material this time! HeteroglossicUsing a safer material would be strongly advisable! MonoglossicUse a safer material this time! Original/NeutralWe keep a factor of safety to avoid reaching yield stress. HeteroglossicIt would make sense to keep a factor of safety to avoid reaching yield stress. MonoglossicKeep a factor of safety to avoid reaching yield stress.

22. Presentation Style – System of Engagement Martin & White, 2005 – Limited exploration of this system in our study 22 Background  Presentation Style  User Study  Idea Generation  Results