1. Should Your MOOC Forum Use a Reputation System? DERRICK COETZEE, ARMANDO FOX, MARTI A. HEARST, BJÖRN HARTMANN UNIVERSITY OF CALIFORNIA, BERKELEY SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 1

2. One-slide summary Goal: Improve learning in MOOCs by improving forums SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 2

3. One-slide summary Goal: Improve learning in MOOCs by improving forums Reputation systems give points to users who post the most useful questions and responses SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 3

4. One-slide summary Goal: Improve learning in MOOCs by improving forums Reputation systems give points to users who post the most useful questions and responses This work: Controlled experiment in a MOOC where one group used a forum with a reputation system and one without SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 4

5. One-slide summary Goal: Improve learning in MOOCs by improving forums Reputation systems give points to users who post the most useful questions and responses This work: Controlled experiment in a MOOC where one group used a forum with a reputation system and one without Results: Posts in the forum with reputation system received more responses and quicker responses, but led to no significant difference in grades/attrition SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 5

6. Outline Motivation and Background MOOC Forums, Reputation Systems SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 6

7. Outline Motivation and Background MOOC Forums, Reputation Systems Experimental Setup Randomized controlled study, MOOC integration SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 7

8. Outline Motivation and Background MOOC Forums, Reputation Systems Experimental Setup Randomized controlled study, MOOC integration Results SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 8

9. Outline Motivation and Background MOOC Forums, Reputation Systems Experimental Setup Randomized controlled study, MOOC integration Results Discussion / Follow-up work SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 9

10. MOOC Forums Dominant form of student interaction and support in MOOCs today “an essential ingredient of an effective online course” (Mak et al 2010) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 10

11. MOOC Forums Dominant form of student interaction and support in MOOCs today Particularly in technical classes, used primarily for answering questions Administrative, particular quiz problems, concepts etc. SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 11

12. Reputation Systems Used by successful Q&A forums on the web StackOverflow Yahoo! Answers Fast high-quality answers (Mamykina et al, CHI 2011) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 12

13. Reputation Systems Used by successful Q&A forums on the web StackOverflow Yahoo! Answers Fast high-quality answers (Mamykina et al, CHI 2011) Users have a public numeric reputation Users receive points when others upvote their questions and/or responses SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 13

14. SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 14

15. SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 15

16. Reputation Systems: Gaining Privileges As users gain points, they acquire more moderator privileges on the site (e.g. edit question, retag question, delete question…) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 16

17. Experimental Setup Deployed in a single software engineering MOOC on edX Randomized controlled trial (aka A/B testing) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 17

18. Experimental Setup Deployed in a single software engineering MOOC on edX Randomized controlled trial (aka A/B testing) Two independent forums Same software, different reputation features enabled SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 18

19. SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 19

20. SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 20

21. Experimental Setup: edX Integration External website linked from edX Authenticated using edX OpenID SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 21

22. Statistics and Data Overview Course had 5985 enrolled and 532 passing students 1101 study participants, 548 in full-featured forum (with reputation system) vs. 553 in basic forum (without) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 22

23. Statistics and Data Overview Course had 5985 enrolled and 532 passing students 1101 study participants, 548 in full-featured forum (with reputation system) vs. 553 in basic forum (without) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 23 Forum with reputation Forum without reputation Total posts 819587 Questions (threads) 115144

24. Results: Improved forum metrics Faster response times Mean of 59 min vs. 2hr20min Mann-Whitney U (p < 0.001) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 24

25. Results: Improved forum metrics Faster response times Mean of 59 min vs. 2hr20min Mann-Whitney U (p < 0.001) More responses per post Mean of 3.5 vs. 2.4 responses Mann-Whitney U (p < 0.001) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 25

26. Results: No significant difference in student outcomes Grades SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 26

27. Results: No significant difference in student outcomes Grades Retention/Attrition SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 27

28. Results: No significant difference in student outcomes Grades Retention/Attrition Rovai’s Sense of Community 20-question survey measuring how much student feels like “I belong to a community that I can trust and depend on” SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 28

29. Results: Reputation System Decreases Appeals for Help Example of analysis based on post content Examples: “Has anyone else experienced something similar?”, “Any help would be greatly appreciated.” Theory: no need to appeal for help if system rewards helpers Looked for posts containing appeals for help Three raters rated posts as appeal for help or not (κ = 0.80) More appeals for help in forum without reputation system 7.3% of posts vs. 2.9% (Fisher’s exact test, p ≈ 0.0002) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 29

30. Do discussion forums affect learning at all? SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 30 Cheng et al, 2011: "an online discussion forum was implemented at the beginning of an undergraduate introductory psychology course [...] students who participated in the forum tended to have better performance in the course, and […] participating in the discussion forum, particularly reading posts on the forum, slightly improved exam performance“ Replicate in MOOC setting?

31. Subjects – Controlled study SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 31 Others (4884) Forum with reputation system (548) Forum without reputation system (553)

32. Subjects – Descriptive/Retrospective study SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 32 Others (1192) Visited any forum at least once (1572) Viewed at least one lecture, HW, or quiz (4793) Non-forum users (3221) Self- selected

33. Results: Forum use predicts better student outcomes Grades SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 33 AssignmentForum users have higher scores? Significance Homework 1Yesp < 0.001 Homework 2Yesp < 0.02 Homework 3Not shownp > 0.1 Homework 4Not shownp > 0.2 Quiz 1Yesp < 0.001 Quiz 2Yesp < 0.001 Quiz 3Yesp < 0.008

34. Grades Retention Median 39 of 42 days, vs. 19 of 42 SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 34 Results: Forum use predicts better student outcomes

35. Discussion Main result: Reputation systems improve forum experience with no evidence of damaging student outcomes, but also no assurance of improved outcomes SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 35

36. Discussion Main result: Reputation systems improve forum experience with no evidence of damaging student outcomes, but also no assurance of improved outcomes Possible explanation for lack of significant difference: low participation SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 36

37. Discussion: Low participation In any given week, 13-17% of students accessing course website accessed experimental forum website SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 37

38. Discussion: Low participation In any given week, 13-17% of students accessing course website accessed experimental forum website About half of subjects accessed 5 or less threads ever Significant differences in outcomes unlikely when most subjects don’t even use the feature Also found low participation in work with chats [L@S 2014] SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 38

39. Limitations SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 39/46 Single course offering (generalizable?) Small number of very active users (see Teo & Johri, this conference) Bias due to consent procedure Contamination (outside interaction)

40. Lessons for field study researchers in MOOCs SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 40 Study retention and performance separately Student who drops after getting 100% on one quiz may have same final grade as a student who gets 25% on every quiz

41. Lessons for field study researchers in MOOCs SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 41 Study retention and performance separately Student who drops after getting 100% on one quiz may have same final grade as a student who gets 25% on every quiz Don’t rely on late surveys – only hits biased survivor group

42. Lessons for field study researchers in MOOCs SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 42 Study retention and performance separately Student who drops after getting 100% on one quiz may have same final grade as a student who gets 25% on every quiz Don’t rely on late surveys – only hits biased survivor group Use established platforms, working around extensibility limitations

43. Follow-up work SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 43 “Chatrooms in MOOCs: All Talk and No Action” (Learning @ Scale 2014) Another controlled experiment Deployed a global synchronous chatroom on every page Low participation; no significant effect on student outcomes

44. Follow-up work SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 44 “Chatrooms in MOOCs: All Talk and No Action” (Learning @ Scale 2014) Another controlled experiment Deployed a global synchronous chatroom on every page Low participation; no significant effect on student outcomes Peer learning chat (in progress) Students discuss questions in synchronous chat in small groups Early work with Turk simulations shows users enjoy using it Planned to be deployed in a MOOC

45. Questions? SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 45 Derrick Coetzee (dcoetzee@berkeley.edu) Armando Fox (fox@berkeley.edu) Marti A. Hearst (hearst@berkeley.edu) Björn Hartmann (bjoern@berkeley.edu) Supported by National Science Foundation under award IIS-1149799 and partially funded by the National Endowment for the Humanities under grant HK-50011