1. Computer Supported Collaborative Learning Track Introduction Carolyn Penstein Rosé Carnegie Mellon University Language Technologies Institute and Human-Computer Interaction Institute School of Computer Science

2. Track Helpers 2 Rohit Kumar rohitk@andrew.cmu.edu Iris Howley iris@cmu.edu

3. Outline Individual learning with technology to collaborative learning Vision for dynamic collaborative learning support Research Issues Current Directions 3

4. Historical Perspective… Socratic tutoring: directed lines of reasoning –Evidence that socratic tutoring is more beneficial than didactic tutoring (Rosé et al., 2001a) Socratic style implemented in Knowledge Construction Dialogues (KCDs) –General attempt to model effective human tutoring –Hierarchical structure: adaptive to student needs –Used to elicit reflection –First used to support individual learning in Physics Atlas-Andes (Rosé et al., 2001) WHY-Atlas (Rosé et al., 2003; Rosé & VanLehn, 2005)

5. Empirical Foundation for CycleTalk Human tutoring not always better than non- interactive support (VanLehn et al., 2007) –Focus shift to capturing what it is about interaction that is effective for instruction Human tutors guide students towards opportunities for reflection (Rosé & Torrey, 2004) Human tutor support by effective tutors is significantly better than hint based support (Rosé et al., 2005)

6. Towards enriching the interaction… Student interaction with dialogue agents lacks elaboration (Rosé et al., 2003) Students expect to behave differently with agents (Rosé & Torrey, 2004) Students can benefit from working with another student, even in the absence of computer scaffolding (Gweon et al., 2006; Kumar et al., 2007) 6

7. VMT-Basilica ** Students learn up to 1.25 standard deviations more when interactive support is provided in the environment.

8. Download tools at: http://www.cs.cmu.edu/~cprose/TagHelper.html http://www.cs.cmu.edu/~cprose/SIDE.html Monitoring Collaboration with Machine Learning Technology TagHelper Labeled Texts Unlabeled Texts Labeled Texts A Model that can Label More Texts Time Behavior <Triggered Intervention>

9. Basilica Tutor Agent Design RequestDetecto r T.TakingCoordinator TutoringManag er OutputCoordinator ConcertChatActorConcertChatListener ConcertChat Server TutoringActor PromptingActor MessageFilter DiscourseMemory PresenceFilter PromptingManage r IntroductionsActor IntroductionsManag er ActivityDetector AnnotationFilter PlanExecutor SocialControlle r ProgressDetector Tutor One last thing on this topic, Does more (or less) stress in a wrench make it easier to use? S95no change? TutorYou are correct about that.. Stress doesn't determine ease of use. Tutor It's the moment achieved by the wrench that determines the ease of use. S89yay! Tutor It's good to have your contributions in the discussion Jackie :-) TutorGo team :-) S89Go team yay Tutor I am happy to be working with our team S89Me too S95whoa the bot knows my name Tutor Unfortunately maximum stress (12800) in our design1 is way above the maximum allowed stress (i.e. 8750) TutorThis wrench cannot be safely used! Kumar, R. & Rosé, C. P. (2011). Architecture for building Conversational Agents that support Collaborative Learning, IEEE Transactions on Learning Technologies special issue on Intelligent and Innovative Support Systems for Computer Supported Collaborative Learning

10. Computer Supported Learning in Large Classroom Studies University Level –Freshman Engineering –Calculus –Sophomore Thermodynamics High School –9 th Grade Biology –Earth Sciences Middle School –Engineering Outreach –6 th grade math

11. First-Year and Middle School Projects First-Year Project –Wrench Design Competition Part of Computer Aided Engineering Labs using Pro/ENGINEER, Pro/MECHANICA and Pro/MANUFACTURING Software (3 sessions) Students Work in Teams of 3 – 4 with Unknown Team Members in Other Parts of the Room Only Communication is via the Chat Interface with Agent Interactions Teach Relationships between Moment, Force and Stress Related to Design Goals Middle-School –Practical Considerations in Designing a Wrench A Simplified Version of the Freshman Project (Using the Pro/ENGINEER CAD Package) Part of the CMU C-MITES Program Students Work in Teams of 3 – 4 with Unknown Team Members in Other Parts of the Room Activity Designed to Elicit Broad Discussions on Engineering and Product Design Middle School Students in the Wrench Design Project

12. Sophomore Thermodynamics Course –Rankine Cycle Design Competition Part of a Thermodynamic Cycles Lab (1 session) Students Work in Teams of 2 with Their Partner in Another Part of the Room Instructed on Effects of Changing System Variables (Temperature, Pressure) on System Output (Power, Waste Heat) Goal: Increase Efficiency within Practical Constraints Software Package is CyclePad (Developed at Northwestern University (Forbes et. al. 1999) to Allow Students to Construct and Analyze a Variety of Thermodynamic Cycles) Second-Year Thermodynamics Project

13. Thermo Collaborative Task Goal: Design a power plant based on the Rankine Cycle paradigm –Each pair turns in exactly one design Competing Student Goals: –Power: Design a power plant that achieves maximum power output Motivated by economic concerns –Green: Design a power plant that has the minimum impact on the environment Motivated by environmental concerns

14. Increasing Conceptual Depth Student: so i think the best thing to go with first is to increase the pressure entering the connector Tutor: You might want to review what you saw earlier in the sensitivity analysis investigating the effect of changing P-max on steam quality and efficiency, from the top half of page 13 in your book. Tutor: type: HELP WITH CHANGEPMAX if you want to discuss it with me. Student: help with changepmax Tutor: Remember the sensitivity analysis you have completed where you manipulated P-Max and observed it's effect on the efficiency of the cycle. You can refer back to Figure 11 on page 13 of your booklet, which …

15. Example Interaction

16. Core Research Focus 16 Identify conversational interactions that are valuable for learning Automatic conversation analysis –Facilitates learning research –Automates assessment of group processes –Enables context sensitive triggering of support Interactive support technologies

17. Operationalization of Productive Talk for Learning 17 Drawn from multiple frameworks valuing sociocognitive engagement –Students explicitly display their reasoning –Students orient their contributions towards previous contributions –Students increase in their ownership of their ideas Made precise through constructs from linguistics –Integrative book chapters (Sionti, Ai, Rosé, & Resnick, in press; Howley, Mayfield, & Rosé, in press)

18. Learning Results Students in accountable talk classrooms (where transactivity is elicited) learn more than students in a traditional classroom (O’Connor, Michaels, & Resnick, in preparation) Transactivity correlates with learning (Joshi & Rosé, 2007; Kumar et al., 2007) –Consistent with results in connection with elaborated explanations (e.g., Webb, Nemer, Zuniga, 2002) Collaboration support that increases transactivity increases learning (Wang et al., 2007) 18

19. Research Question: Tutorial dialogue agents are effective for individual learning However! –Students appear to ignore tutorial dialogue agents when another human is in the environment Although they learn from them! –Dialogue agents have been seen as an interruption in collaborative contexts How must the design of intelligent tutoring technology change in order to be effective in collaborative contexts?

20. Accountable Talk (O’Connor, Michaels, & Resnick) Eddie: Well, i don't think it matters what order the numbers are in. You still get the same answer. But three times four and four times three seem like they could be talking about different things. Teacher: Rebecca, do you agree or disagree with what Eddie is saying? Rebecca: Well, I agree that it doesn't matter which number is first, because they both give you twelve. But I don't get what Eddie means about them saying different things. Teacher: Eddie, would you explain what you mean? Eddie: Well, I just think that like three times four can mean three groups of four things, like three bags of four apples. And four times three means four bags of three apples, and those don't seem like the same thing. Tiffany: But you still have the same number of apples, so they are the same! Teacher: OK, so we have two different ideas here to talk about. Eddie says the order does matter, because the two orders can be used to describe different situations. So Tiffany, are you saying that three times four and four times three can't be used to describe two different situations?

21. Supporting Accountable Talk Research question: What form of support for small group discussion is most effective: –Elevating quality of small group discussion –Learning during small group discussion –Preparation for whole group discussion Instruction: Students read about Diffusion and receive training on Accountable Talk Online Lab in ConcertChat environment –Videos alternate with small group discussion Students watch experimental setup and then predict outcomes Students watch results at 1 hour, 5 hours, and 24 hours and then discuss whether effects matched predictions and what they learned 21

22. 3 students per group –Each student assigned a role Revoicer: Responsible for looking for revoicing opportunities Challenger: Responsible for looking for opportunities to check agreement or challenge a claim Explainer: Responsible for looking for opportunities to push for more explication –Each group assigned a condition No support: students just assigned roles Indirect Agent: agent reminds students to do their role Direct Agent: agent does accountable talk moves 22

23. Accountable Talk Agents 23

24. Results from Coded Chats Significantly more Academically Productive Talk moves in supported conditions –F(2,42) = 13.9, p <.0001 –Weak correlation between Academically Productive Talk moves and student reasoning, R 2 =.11, p <.05 Students in Direct contribute marginally more reasoning than Indirect –F(2,42) = 2.46, p <.1 –Significant when we consider percentage of reasoning moves, F(2,42) = 4.47, p <.05 24 ConditionAcademically Productive Talk Moves Academically Productive Talk Moves Group + Tutor Reasoning Moves Transactive Moves Unsupported.56 (2.7%)1.6 (1.8%)1.6 (11%).55 (2.7%) Indirect Agent1.2 (4.9%)3.8 (3.6%).53 (3.8%).13 (1.1%) Direct Agent.67 (6.4%)4.25 (7%)2 (17%).92 (5.1%)

25. Current Directions Continuing to investigate social considerations for integrating dialogue agents with groups –Investigating how motivation orientation interacts with treatment Working with groups larger than pairs (Gweon et al., in press) –Monitoring collaboration quality from speech –Challenges of multi-party conversation analysis Multiple interwoven threads (Rosé et al., 1995; Wang et al., 2008a,b) 25

26. 26 Thank You !!! Funding: The Office of Naval Research and the National Science Foundation