1. Livenotes A System for Cooperative and Augmented Note-Taking in Lectures Matthew Kam, Jingtao Wang, Alastair Iles, Eric Tse, Jane Chiu, Daniel Glaser, Orna Tarshish and John Canny University of California, Berkeley, USA

2. Video

3. Outline Motivation Solution Experiment Results Implications Conclusion

4. Motivation: Problem Statement Constructivism Learners are not blank slates that teachers write on Learners need to actively construct their own understanding and knowledge But large lecture classes are not conducive for active learning Passive mode of oral dissemination Lack of interactivity among students Lack of interactivity with instructor

5. Motivation: Precedents Face-to-Face Tutored Video Instruction (TVI), Distributed TVI (Gibbons, Stanford) Group review of pre-recorded lectures Regular pauses for small-group discussion Students using DTVI received grades 0.5 std dev higher than non-TVI students (Smith et al. 1999) Peer Instruction (Mazur, Harvard) Lecture pauses for small-group discussion with neighbors Improvements in conceptual understanding and problem-solving (Crouch and Mazur 2001)

6. Motivation: Small-Group, Cooperative Learning More than 375 research studies since 1898 (Johnson and Johnson 1989) Cooperative group learning results in greater Efforts to achieve Higher-level reasoning Transfer from original context to new situations Generation of new ideas and solutions

7. Motivation: Background Lecture Notes (Hartley 1978, Kiewra et al. 1988) Experiments on note-taking that compared students annotating over: 1) Complete lecture notes provided by instructor, vs. 2) Skeletal (i.e. partial) notes, vs. 3) No background notes Results: students were found to achieve maximum retention with skeletal notes

8. Livenotes Recap Both a technology and educational practice Large lecture classes Small-group discussions in ongoing lecture Cooperative note-taking: Combines real-time note-taking with discussion Augmented note-taking: Skeletal slides for students to annotate over

9. Related Systems No interaction between students Classroom Presenter (Washington) StuPad, eClass (Georgia Tech) No real-time interaction between students, i.e. sharing of notes takes place after lecture NotePals (Berkeley) Limited real-time interaction between students OneNote (Microsoft)

10. Outline Motivation Solution Experiment Results Implications Conclusion

11. Livenotes Evolution 2000: Implemented in Java, for WinCE Clios Late 2000 to early 2003: 5 small-scale deployments using Clios, laptops and Tablet PCs Spring 2003: Medium-scale experiment in undergraduate class using Tablet PCs Since 2003: Ported to Microsoft.NET

12. Livenotes User Interface Pen and keyboard input Group awareness (e.g. each user’s page number) Import background slides Unique user colors

13. Client-Server Topology 802.11b networking Large class broken down into many small groups (3-7 students) One Tablet per group is set to server mode Other members’ Tablets connect wirelessly to group’s server Server Clients Server Clients Group n: Group 1: …

14. Outline Motivation Solution Experiment Results Implications Conclusion

15. Hypotheses Cooperative note-taking: Shared whiteboard interface enhances learning through cooperative note-taking and discussion Augmented note-taking: Background slides enhances learning by augmenting student note-taking

16. Experiment Spring 2003 undergraduate HCI class 21 volunteers, randomly partitioned into Cooperative note-takers Individual note-takers (control group) 4 weeks (7 lectures) Preloaded skeletal PowerPoint slides

17. Previous Observation From 5 previous deployments, we learned that Graduate students engaged spontaneously in group discussions Undergraduates were not used to discussing lecture material with one another For this experiment (with undergraduates), we held short, live group discussions in the classroom

18. Data Collection Short quizzes (4 lectures) Survey questionnaires First week of deployment (~38% response rate) End of semester (~29%) Qualitative interviews (3 users) Transcripts of students’ notes (~1581 pages)

19. Quantitative Analysis Unit of analysis: mark Spatio-temporally contiguous segment of user input E.g.: “This lecture is very interesting” Quantitative hand- coding of ~1581 pages

20. Taxonomy of Marks Note-taking: someone taking notes on lecture Commentary: someone making a statement Question: someone soliciting a response Answer: response to a question, clarification Reinforcement:contribution to an existing thread

21. Outline Motivation Solution Experiment Results Implications Conclusion

22. Cooperative Note-Taking: Richer Notes Cooperative note-taking group engaged in more than twice as much activity as individual note-taker

23. Cooperative Note-Taking: Richer Notes Almost one quarter of marks made by cooperative note-takers were attributed to group interaction

24. Student Learning Survey question: “How did Livenotes, if at all, assist your learning in lecture?” Early survey after 2 sessions with Livenotes: 75% of respondents self-reported affirmatively Survey after semester (i.e. 7 lecture sessions): 83% of respondents self-reported affirmatively

25. Cooperative Note-Taking: Taking Turns to Take Notes 66% of survey respondents agreed that cooperative note-taking is more useful “Someone else might note something that I missed or hadn’t realized.” “I liked how note-taking became a cooperative effort … someone can take over if another user is still inputing some notes, but the prof [had] moved on already.”

26. Cooperative Note-Taking: Paying Greater Attention 36% of students who self-reported learning benefit explicitly attributed that to social aspect of cooperative note-taking: “Helped me to focus more in lecture. Often I fall asleep/lose attention in lecture. Having group members to respond to kept me better on track.”

27. Cooperative Note-Taking: Dual Conversations Need to keep up with both lecture and on- tablet conversation: “It is helpful to be able to discuss questions. However, this does take attention away from the lecture if you are focusing on answering/asking a question.”

28. Cooperative Note-Taking: Decreasing Distraction Is “running Livenotes during class distracting?” (1 = extremely distracting, 5 = not distracting at all) Survey after two lectures: 2.6 out of 5 Survey after deployment ended: 3.83 out of 5 From student notes, “playful” behavior were observed to disappear almost completely after 2 lectures

29. Cooperative Note-Taking: Unanswered Group Questions Students did not have time to answer some questions because they needed to keep up with lecturer Some questions were unanswered because no group member knew the answer

30. Cooperative Note-Taking: Interaction During Pauses Group interaction during pauses in lecture accounted for over half of group activity

31. Redeeming PowerPoint Criticisms leveled at Microsoft PowerPoint The Cognitive Style of PowerPoint (Edward Tufte) Death by PowerPoint webpage I Powerpoint (David Byrne, Talking Heads) Too boring, passive, does not promote active engagement with material But students commented that augmented note-taking is “like having a conversation with the professor”

32. Augmented Note-Taking: Observed Behaviors Elaborated on bullets Appended bullets to list Concurred and disagreed with bullet Noted gist of HCI principles Noted advantage and disadvantage of HCI technique Answered questions in bullets

33. Augmented Note-Taking: Elaborated on Bullets

34. Augmented Note-Taking: Appended Bullets to List

35. Augmented Note-Taking: Answered Questions in Bullet

36. Students responded to questions in bullets even when when they were not cooperative note-takers Each group responded to 35% of the questions Each question received a response from 36% of the groups

37. Augmented Note-Taking: Student Learning Several high-quality notes in both individual and cooperative note-taking groups resulted from students “working off” bullets Possibly due to bullets focusing student attention to relevant portions of lecture A larger proportion (55%) of students who self- reported learning benefit attributed it to augmented note-taking, compared to cooperative note-taking. Half of this sub-group attributed that to having slides at hand to annotate over

38. Quiz Scores Quiz 1Quiz 2Quiz 3Quiz 4 C [1]I [2]CICICI Mean Score [3] 63.966.071.275.058.567.453.255.7 Std. Dev. 9.27.312.1n/a9.511.58.624.4 Sample size 84919474 P-value 0.7020.7710.1680.810 No statistical significance But sample size was too small due to poor attendance at end of semester [1] Cooperative note-takers. [2] Individual note-takers. [3] Quiz scores presented in this table are normalized on a scale of 100.

39. Outline Motivation Solution Experiment Results Implications Conclusion

40. Student-Instructor Interaction To help instructor assess student learning, we deployed feedback feature in last two sessions Students provide instructor with real-time, anonymous lecture feedback Recently allowed students to alert instructor that they have questions

41. Recommendation: Background Slides as Scaffold Bullets are a lightweight means for lecturer to engage actively with students during class Posing questions Counter-intuitive bullets Provocative statements Direct student attention to critical parts of lecture E.g. prompts such as “Pros?” and “Cons?” with blank spaces for students to fill in

42. Outline Motivation Solution Experiment Results Implications Conclusion

43. Cooperative note-taking Richer variety of notes, higher-order thinking More than twice as much notes as individuals Members took turns to take notes Students kept awake to interact with group Augmented note-taking Observed dialogue with bullets Reflected higher-order thinking High-quality notes resulted from “working off” bullets

44. Acknowledgement Corporation for Education Network Initiatives in California Microsoft Research National Science Foundation Qualcomm Volunteers from Computer Science 160, Spring 2003 Public domain source code by James R. Weeks

45. Questions? Livenotes can be downloaded from: http://www.cs.berkeley.edu/~mattkam/livenotes Matthew Kam, Ph.D. student Department of Electrical Engineering and Computer Sciences, and Berkeley Institute of Design University of California at Berkeley, USA mattkam@eecs.berkeley.edu