Carolyn Penstein Rosé Language Technologies Institute Human-Computer Interaction Institute School of Computer Science With funding from the National Science Foundation and the Office of Naval Research 1

2 Developing technology capable of shaping conversation and supporting effective participation in conversation to achieve positive impact on… Human learning Health Wellbeing

3 Developing technology capable of shaping conversation and supporting effective participation in conversation to achieve positive impact on… Human learning Health Wellbeing Human learning

4 Introducing the Problem of Supporting Productive Discussion for Learning Discussion of Souflé Transactivity Engagement Authoritativeness Application to Dynamic Support for Group Learning Conclusion and Current Directions Outline

5 Introducing the Problem of Supporting Productive Discussion for Learning Discussion of Souflé Transactivity Engagement Authoritativeness Application to Dynamic Support for Group Learning Conclusion and Current Directions Outline

 Reward structures encourage students to focus on performance over learning  Well crafted instruction provides opportunities for learning  Opportunities only help if students take them Take Home Message: Introducing reflection points provides opportunities for students to take advantage of learning resources

Carefully Structured Conceptual Knowledge 7 Reflection through Rich Interaction

 End of Fall Semester: Students learn about Rankine Cycles  1 Week of lectures  Homework assignment on analysis of Rankine Cycles  Tutorial on using CyclePad software package (Developed at Northwestern University (Forbes et. al. 1999)  Allows students to construct and analyze a variety of Thermodynamic Cycles)  Instructed on Effects of Changing System Variables (Temperature, Pressure) on System Output (Power, Waste Heat)

 Learning Goal: Encourage students to reflect on interactions between cycle parameters Reduction in Steam Quality Power Waste Heat Increasing heat increases power but also waste heat Increasing pressure increases efficiency  Design Goal: Design a power plant based on the Rankine Cycle paradigm  Competing Student Goals:  Power: Design a power plant that achieves maximum power output  Motivated by economic concerns  Green: Design a power plant that has the minimum impact on the environment  Motivated by environmental concerns  Each pair turns in exactly one design

11

12 Introducing the Problem of Supporting Productive Discussion for Learning Discussion of Souflé Transactivity Engagement Authoritativeness Application to Dynamic Support for Group Learning Conclusion and Current Directions Outline

13 Person 3 Dimensions:  Transactivity  Engagement  Authoritativeness

14 Sociolinguistics Discourse Analysis Language And Identity Language Use Machine Learning Multi- Level Modeling Applied Statistics Computational Models Of Discourse Analysis

15 Transactive Knowledge Integration Person

16 Definition of Transactivity building on an idea expressed earlier in a conversation using a reasoning statement We don't want tmax to be at 570 both for the material and [the Environment] well, for power and efficiency, we want a high tmax, but environmentally, we want a lower one.

17

18

19

 Findings  Moderating effect on learning (Joshi & Rosé, 2007; Russell, 2005; Kruger & Tomasello, 1986; Teasley, 1995)  Moderating effect on knowledge sharing in working groups (Gweon et al., 2011)  Computational Work  Can be automatically detected in:  Threaded group discussions (Kappa.69) (Rosé et al., 2008)  Transcribed classroom discussions (Kappa.69) (Ai et al., 2010)  Speech from dyadic discussions (R =.37) (Gweon et al., 2012)  Predictable from a measure of speech style accommodation computed by an unsupervised Dynamic Bayesian Network (Jain et al., 2012) Transactivity (Berkowitz & Gibbs, 1983)

Engagement 21 Transactive Knowledge Integration Person

 System of Engagement  Showing openness to the existence of other perspectives  Less final / Invites more discussion  Example:  [M] Nuclear is a good choice  [HE] I consider nuclear to be a good choice  [HC] There’s no denying that nuclear is a superior choice  [NA] Is nuclear a good choice? 22

23

 Findings  Correlational analysis: Strong correlation between displayed openness of group members and articulation of reasoning (R =.72) (Dyke et al., in press)  Intervention study: Causal effect on propensity to articulate ideas in group chats (effect size.6 standard deviations) (Kumar et al., 2011)  Mediating effect of idea contribution on learning in scientific inquiry (Wang et al., 2011) Engagement (Martin & White, 2005)

Authority Engagement 25 Transactive Knowledge Integration Person

Analysis of Authoritativess 26 Water pipe analogy: Water = Knowledge or Action Source = Authoritative speaker Sink = Non-authoritative Speaker Authoritativeness Ratio = Source Actions Actions

Source or Sink? Primary Secondary Type of Content? Knowledge Action K2 requesting knowledge, information, opinions, or facts K1 giving knowledge, information, opinions, or facts A2 Instructing, suggesting, or requesting non-verbal action A1 Narrating or performing your own non-verbal action Additionally… ch (direct challenge to previous utterance) o (all other moves, backchannels, etc.) 27

Source or Sink? Primary Secondary Type of Content? Knowledge Action K2 requesting knowledge, information, opinions, or facts K1 giving knowledge, information, opinions, or facts A2 Instructing, suggesting, or requesting non-verbal action A1 Narrating or performing your own non-verbal action 28

Source or Sink? Primary Secondary Type of Content? Knowledge Action K2 requesting knowledge, information, opinions, or facts K1 giving knowledge, information, opinions, or facts A2 Instructing, suggesting, or requesting non-verbal action A1 Narrating or performing your own non-verbal action 29

Source or Sink? Primary Secondary Type of Content? Knowledge Action K2 requesting knowledge, information, opinions, or facts K1 giving knowledge, information, opinions, or facts A2 Instructing, suggesting, or requesting non-verbal action A1 Narrating or performing your own non-verbal action

Source or Sink? Primary Secondary Type of Content? Knowledge Action K2 requesting knowledge, information, opinions, or facts K1 giving knowledge, information, opinions, or facts A2 Instructing, suggesting, or requesting non-verbal action A1 Narrating or performing your own non-verbal action Additionally… ch (direct challenge to previous utterance) o (all other moves, backchannels, etc.) 31

Source or Sink? Primary Secondary Type of Content? Knowledge Action K2 requesting knowledge, information, opinions, or facts K1 giving knowledge, information, opinions, or facts A2 Instructing, suggesting, or requesting non-verbal action A1 Narrating or performing your own non-verbal action Additionally… ch (direct challenge to previous utterance) o (all other moves, backchannels, etc.) 32 K1 + A2 K1 + K2 + A1 + A2 Authoritativeness:

33 K2?

34 Set up! K1 K2

35

 Findings  Authoritativeness measures display how students respond to aggressive behavior in groups (Howley et al., in press)  Authoritativeness predicts learning (R =.64) and self-efficacy (R =.35) (Howley et al., 2011)  Authoritativeness predicts trust in doctor-patient interactions (R values between.25 and.35) (Mayfield et al., under review)  Computational Work  Detectable in collaborative learning chat logs (R =.86)  Detectable in transcribed dyadic discussions in a knowledge sharing task (R =.95) (Mayfield & Rosé, 2011)  Detectable in transcribed doctor-patient interactions (R =.96) (Mayfield et al., under review) Authoritativeness (Martin & Rose, 2003)

37 Introducing the Problem of Supporting Productive Discussion for Learning Discussion of Souflé Transactivity Engagement Authoritativeness Application to Dynamic Support for Group Learning Conclusion and Current Directions Outline

39 Automatic Analysis Of Conversation Conversational Interventions Positive Learning Outcomes

 Foundational study: students work with a partner and dialogue agent for support Learn 1.24 s.d. more than individuals without support (Kumar et al., 2007a)  Results inform iterative design of agent behavior  Personalized agents increase supportiveness and help exchange between students (Kumar et al., 2007b)  Agents are more effective when students have control over timing of the interaction (Chaudhuri et al., 2008; Chaudhuri et al., 2009)  Agents that employ Balesian social strategies are more effective than those that do not (Kumar et al., 2010; Ai et al., 2010)  Students are sensitive to agent rhetorical strategies such as displayed bias (Ai et al., 2010), displayed openness to alternative perspectives (Kumar et al., 2011), and targeted elicitation (Howley et al., 2012)  Bazaar architecture enables efficient, principle based agent development (Kumar & Rosé, 2011; Adamson & Rosé, 2012)

41 Introducing the Problem of Supporting Productive Discussion for Learning Discussion of Souflé Transactivity Engagement Authoritativeness Application to Dynamic Support for Group Learning Conclusion and Current Directions Outline

42 Transactivity is a conversational behavior that is important for learning Authoritativeness and Engagement are dimensions of conversation that play a supporting role Positioning students to exchange Transactive contributions We have made progress at automating analysis of Transactivity and Authoritativeness Automated analysis enables dynamic triggering of supportive interventions for online group learning Conclusions and Current Directions

43 In the future, CSCL activities will be part of societies of online learners Vision began with Virtual Math Teams/ The Math Forum Now we’re already seeing the shift through companies like Coursera and Udacity We are taking steps towards this future Fully distance learning studies (UCSB, Drexel) Sustainable CSCL (online office hours agent) As we look to the future: we must understand the emergent effects of our local interventions in order to maximize positive benefit on a grand scale Conclusions and Current Directions

44

45

 Experimental Paradigm  Day 1: Pretest on conceptual questions related to the unit (Diffusion or Punnett Squares)  Day 2: Online collaborative activity + Immediate Posttest isomorphic to pretest  Day 3: Whole class teacher led discussion + Delayed Posttest isomorphic to pretest  Participants: consenting 9 th grade biology students, randomly assigned to groups of 3  Experimental Design: Simple between subjects design  Groups randomly assigned to Revoicing condition or Control Condition 46

47

 Study 1: Year 1, Diffusion Lab (50 students)  Students learn more on explanation questions in supported conditions (F(1,46) = 4.3, p <.05, effect size 1 standard deviation)  Students in supported conditions more active in whole group discussion (F(2,26) = 4.2, p <.05, effect size.75 standard deviations)  Study 2: Year 2, Diffusion Lab (78 students)  Students learn more on immediate post test in Revoicing Agent condition (F(1,74) = 4.3, p <.05, effect size.51 standard deviations)  Study 3: Year 2, Punnett Square Lab (78 students)  Students learned more on delayed post-test in Revoicing Agent condition 48