Should Your MOOC Forum Use a Reputation System? DERRICK COETZEE, ARMANDO FOX, MARTI A. HEARST, BJÖRN HARTMANN UNIVERSITY OF CALIFORNIA, BERKELEY SHOULD.

Slides:



Advertisements
Similar presentations
Esri International User Conference | San Diego, CA Demo Theater | ArcGIS Beta Community and ArcGIS 10.1 Beta Program Mike Hogan & Rohit Gupta July 12 th,
Advertisements

Agent-Based Architecture for Intelligence and Collaboration in Virtual Learning Environments Punyanuch Borwarnginn 5 August 2013.
Register Laulima Workshop for Instructors Solutions to help you engage your students through Laulima.
Access and Excellence: MOOCs & Online Education at UC Berkeley Armando Fox, Academic Director, UC Berkeley MOOCLab 1.
Today’s Agenda  Syllabus CS2336: Computer Science II.
Chatrooms in MOOCs: All Talk and No Action DERRICK COETZEE, ARMANDO FOX, MARTI A. HEARST, BJÖRN HARTMANN UNIVERSITY OF CALIFORNIA, BERKELEY.
Role of Vendor Technologies in the Development of Network Professionals Mak Sharma and Sharon Cox School of Computing, Telecommunications and Networks.
Continuous Improvement in Teaching and Learning Candace Thille Director, Open Learning Initiative.
Educational portal and its integration with AHyCo LMS Nataša Hoić–Božić University of Rijeka (Croatia)
Engaged learning in MOOCS Julie Wintrup With research team: Kelly Wakefield, Debra Morris, Hugh Davis Funded by HEA.
CS/CMPE 535 – Machine Learning Outline. CS Machine Learning (Wi ) - Asim LUMS2 Description A course on the fundamentals of machine.
Evaluation Methodologies
Graduate Student Satisfaction with an Online Discrete Mathematics Course Amber Settle, CTI, DePaul University joint work with Chad Settle, University of.
IMPLEMENTATION OF AN E-LEARNING PLATFORM USING CMS
IACT303 – INTI 2005 World Wide Networking Welcome and Introduction to Subject. Penney McFarlane The University of Wollongong.
Online Respiratory Care Curriculum.  Navigate Respiratory Care: Cardiopulmonary Anatomy & Physiology  Navigate Respiratory Care: Pharmacology  Navigate.
COURSE NAME & # Redesigning Introductory Spanish: A Blended Approach Glynis Cowell, Director, Spanish Language Program Hosun Kim, Director, Foreign Language.
V v Adaptive Quizzing Associated with an Increase in Overall Learning Keiko Bostwick, Kathryn Becker-Blease, PhD Oregon State University School of Psychological.
International Conference on Enhancement and Innovation in Higher Education Crowne Plaza Hotel, Glasgow 9-11 June 2015 Welcome.
Psychology students’ experiences of peer tutoring in academic writing European Congress of Psychology, Oslo, July 2009 Kathy Harrington, Savita Bakhshi.
CA12 Assessing Online Courses Howard University November 2013.
Can MOOCs Reduce Teaching Costs and Provide High Quality Undergraduate Education? High School Students Undergraduate Education Careers, Employment Uday.
Chris Evans, University of Winchester Dr Paul Redford, UWE Chris Evans, University of Winchester Dr Paul Redford, UWE Self-Efficacy and Academic Performance:
MOOC as a Learning Environment and its Educational Values Abeer Watted and Miri Barak
Southern Regional Education Board HSTW Ten Conditions that Result in Students Making Greater Effort to Succeed Gene Bottoms.
E- Communication An Introduction to On-Line Forums in the Classroom Presented by Leanna Prater and Jeff Jones District Technology Resource Teachers Fayette.
ThomsonNOW on Sakai Dan Rinn Thomson Higher Education Manager, Technology Partnership Marketing.
the best method of teaching
Online Student Success:
Jay Summet CS 1 with Robots IPRE Evaluation – Data Collection Overview.
FITT Fostering Interregional Exchange in ICT Technology Transfer Communication & Collaboration Tools.
Introduction to Moodle CAK Inservice. Don’t forget: You can copy- paste this slide into other presentations, and move or resize the poll.
Introduction to Science Informatics Lecture 1. What Is Science? a dependence on external verification; an expectation of reproducible results; a focus.
SS440 Seminar: Unit 4 Research in Psychopathology Dr. Angie Whalen Kaplan University 1.
Towards Collaborative Scale. 20 million minds foundation.
WebCT: Managing Students and Communicating FITC Staff.
Who is on… Introduction Using social media entails particular kind of literacies i.e. skills which include the ability to engage in a medium for production.
Learning Management Systems A Review and Recommendation of three different LMS By Carissa Boulton.
Student Preferences For Learning College Algebra in a Web Enhanced Environment Dr. Laura J. Pyzdrowski, Pre-Collegiate Mathematics Coordinator Institute.
Assessment in Online Courses Gerd Kortemeyer Michigan State University APS/AAPT 2013 Department Chairs Conference and Workshop Workshop Access
FORESTUR How to work… …with this training platform? …with this methodology?
Technology in the Classroom: A Working Discussion Group Nelson C. Baker, Ph.D. Georgia Tech SUCCEED College of Engineering CETL, OIT-Educational Technologies.
Moodle for Your Noodle. What in the world is Moodle? An open source Content Management System (or Course Management System) built around a sound educational.
The Impact of Student Self-e ffi cacy on Scientific Inquiry Skills: An Exploratory Investigation in River City, a Multi-user Virtual Environment Presenter:
Developing a Metric for Evaluating Discussion Boards Dr. Robin Kay University of Ontario Institute of Technology 2 November 2004.
SCIENTIFIC METHOD RESEARCH METHODS ETHICS PSYCHOLOGICAL RESARCH.
© 2010 Blackboard Inc. All rights reserved. Blackboard Learn 9.1 Making the Most of Discussions.
Research methods revision The next couple of lessons will be focused on recapping and practicing exam questions on the following parts of the specification:
Reflections, Discussion Threads and Peer Review for Assessment in Online Learning Kristine Rabberman, Ph.D. Carol A. Muller, Ph.D.
LAMS: The Learning Activity Management System James Dalziel Professor of Learning Technology & Director, Macquarie E-Learning Centre Of Excellence (MELCOE)
1.On smart phone or laptop: please go to to fill out your contact info. See if anyone else needs to use laptop when you are finished.
AN EXPERIENCE REPORT ASSESSING A PROFESSIONAL DEVELOPMENT MOOC FOR CS PRINCIPLES SIGCSE 2017 – MEMPHIS, TN JEFF GRAY DEPARTMENT OF COMPUTER SCIENCE UNIVERSITY.
International Qualitative Conference STUDENT ACHIEVEMENT USING WEB 2.0 TECHNOLOGIES: A MIXED METHODS STUDY By: Dallas R. Malhiwsky.
NACS Canvas Training Please sign in and log onto a computer.
Education eLibrary and Repository
LAMS: The Learning Activity Management System
Prof. Burks Oakley II Assoc. Vice President for Academic Affairs
IST256 : Applications Programming for Information Systems
Presented to IEEE Standards Education Committee 11 April 2014
MathXL® Student Overview QRB/501
E-learning system "Carat"
Respiratory Therapy Program
MyMathLab® Student Overview QRB/501
Student Overview.
IIC Academy Hydrographic Training Program [Global Delivery] Véronique Jégat Manager Bathymetric Projects IIC Technologies Inc. Derrick Peyton CEO,
MyStatLab Student Overview QNT/561
The Flipped Classroom Model: A Full Picture
Evaluating the Impact of Peer Programmes on Student Learning and Academic Attainment – A Cross-Discipline Approach Presenters: Amanda Pocklington, Bella.
Presentation transcript:

Should Your MOOC Forum Use a Reputation System? DERRICK COETZEE, ARMANDO FOX, MARTI A. HEARST, BJÖRN HARTMANN UNIVERSITY OF CALIFORNIA, BERKELEY SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 1

One-slide summary Goal: Improve learning in MOOCs by improving forums SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 2

One-slide summary Goal: Improve learning in MOOCs by improving forums Reputation systems give points to users who post the most useful questions and responses SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 3

One-slide summary Goal: Improve learning in MOOCs by improving forums Reputation systems give points to users who post the most useful questions and responses This work: Controlled experiment in a MOOC where one group used a forum with a reputation system and one without SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 4

One-slide summary Goal: Improve learning in MOOCs by improving forums Reputation systems give points to users who post the most useful questions and responses This work: Controlled experiment in a MOOC where one group used a forum with a reputation system and one without Results: Posts in the forum with reputation system received more responses and quicker responses, but led to no significant difference in grades/attrition SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 5

Outline Motivation and Background MOOC Forums, Reputation Systems SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 6

Outline Motivation and Background MOOC Forums, Reputation Systems Experimental Setup Randomized controlled study, MOOC integration SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 7

Outline Motivation and Background MOOC Forums, Reputation Systems Experimental Setup Randomized controlled study, MOOC integration Results SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 8

Outline Motivation and Background MOOC Forums, Reputation Systems Experimental Setup Randomized controlled study, MOOC integration Results Discussion / Follow-up work SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 9

MOOC Forums Dominant form of student interaction and support in MOOCs today “an essential ingredient of an effective online course” (Mak et al 2010) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 10

MOOC Forums Dominant form of student interaction and support in MOOCs today Particularly in technical classes, used primarily for answering questions Administrative, particular quiz problems, concepts etc. SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 11

Reputation Systems Used by successful Q&A forums on the web StackOverflow Yahoo! Answers Fast high-quality answers (Mamykina et al, CHI 2011) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 12

Reputation Systems Used by successful Q&A forums on the web StackOverflow Yahoo! Answers Fast high-quality answers (Mamykina et al, CHI 2011) Users have a public numeric reputation Users receive points when others upvote their questions and/or responses SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 13

SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 14

SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 15

Reputation Systems: Gaining Privileges As users gain points, they acquire more moderator privileges on the site (e.g. edit question, retag question, delete question…) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 16

Experimental Setup Deployed in a single software engineering MOOC on edX Randomized controlled trial (aka A/B testing) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 17

Experimental Setup Deployed in a single software engineering MOOC on edX Randomized controlled trial (aka A/B testing) Two independent forums Same software, different reputation features enabled SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 18

SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 19

SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 20

Experimental Setup: edX Integration External website linked from edX Authenticated using edX OpenID SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 21

Statistics and Data Overview Course had 5985 enrolled and 532 passing students 1101 study participants, 548 in full-featured forum (with reputation system) vs. 553 in basic forum (without) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 22

Statistics and Data Overview Course had 5985 enrolled and 532 passing students 1101 study participants, 548 in full-featured forum (with reputation system) vs. 553 in basic forum (without) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 23 Forum with reputation Forum without reputation Total posts Questions (threads)

Results: Improved forum metrics Faster response times Mean of 59 min vs. 2hr20min Mann-Whitney U (p < 0.001) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 24

Results: Improved forum metrics Faster response times Mean of 59 min vs. 2hr20min Mann-Whitney U (p < 0.001) More responses per post Mean of 3.5 vs. 2.4 responses Mann-Whitney U (p < 0.001) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 25

Results: No significant difference in student outcomes Grades SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 26

Results: No significant difference in student outcomes Grades Retention/Attrition SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 27

Results: No significant difference in student outcomes Grades Retention/Attrition Rovai’s Sense of Community 20-question survey measuring how much student feels like “I belong to a community that I can trust and depend on” SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 28

Results: Reputation System Decreases Appeals for Help Example of analysis based on post content Examples: “Has anyone else experienced something similar?”, “Any help would be greatly appreciated.” Theory: no need to appeal for help if system rewards helpers Looked for posts containing appeals for help Three raters rated posts as appeal for help or not (κ = 0.80) More appeals for help in forum without reputation system 7.3% of posts vs. 2.9% (Fisher’s exact test, p ≈ ) SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 29

Do discussion forums affect learning at all? SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 30 Cheng et al, 2011: "an online discussion forum was implemented at the beginning of an undergraduate introductory psychology course [...] students who participated in the forum tended to have better performance in the course, and […] participating in the discussion forum, particularly reading posts on the forum, slightly improved exam performance“ Replicate in MOOC setting?

Subjects – Controlled study SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 31 Others (4884) Forum with reputation system (548) Forum without reputation system (553)

Subjects – Descriptive/Retrospective study SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 32 Others (1192) Visited any forum at least once (1572) Viewed at least one lecture, HW, or quiz (4793) Non-forum users (3221) Self- selected

Results: Forum use predicts better student outcomes Grades SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 33 AssignmentForum users have higher scores? Significance Homework 1Yesp < Homework 2Yesp < 0.02 Homework 3Not shownp > 0.1 Homework 4Not shownp > 0.2 Quiz 1Yesp < Quiz 2Yesp < Quiz 3Yesp < 0.008

Grades Retention Median 39 of 42 days, vs. 19 of 42 SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 34 Results: Forum use predicts better student outcomes

Discussion Main result: Reputation systems improve forum experience with no evidence of damaging student outcomes, but also no assurance of improved outcomes SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 35

Discussion Main result: Reputation systems improve forum experience with no evidence of damaging student outcomes, but also no assurance of improved outcomes Possible explanation for lack of significant difference: low participation SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 36

Discussion: Low participation In any given week, 13-17% of students accessing course website accessed experimental forum website SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 37

Discussion: Low participation In any given week, 13-17% of students accessing course website accessed experimental forum website About half of subjects accessed 5 or less threads ever Significant differences in outcomes unlikely when most subjects don’t even use the feature Also found low participation in work with chats 2014] SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 38

Limitations SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 39/46 Single course offering (generalizable?) Small number of very active users (see Teo & Johri, this conference) Bias due to consent procedure Contamination (outside interaction)

Lessons for field study researchers in MOOCs SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 40 Study retention and performance separately Student who drops after getting 100% on one quiz may have same final grade as a student who gets 25% on every quiz

Lessons for field study researchers in MOOCs SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 41 Study retention and performance separately Student who drops after getting 100% on one quiz may have same final grade as a student who gets 25% on every quiz Don’t rely on late surveys – only hits biased survivor group

Lessons for field study researchers in MOOCs SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 42 Study retention and performance separately Student who drops after getting 100% on one quiz may have same final grade as a student who gets 25% on every quiz Don’t rely on late surveys – only hits biased survivor group Use established platforms, working around extensibility limitations

Follow-up work SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 43 “Chatrooms in MOOCs: All Talk and No Action” Scale 2014) Another controlled experiment Deployed a global synchronous chatroom on every page Low participation; no significant effect on student outcomes

Follow-up work SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 44 “Chatrooms in MOOCs: All Talk and No Action” Scale 2014) Another controlled experiment Deployed a global synchronous chatroom on every page Low participation; no significant effect on student outcomes Peer learning chat (in progress) Students discuss questions in synchronous chat in small groups Early work with Turk simulations shows users enjoy using it Planned to be deployed in a MOOC

Questions? SHOULD YOUR MOOC FORUM USE A REPUTATION SYSTEM? 45 Derrick Coetzee Armando Fox Marti A. Hearst Björn Hartmann Supported by National Science Foundation under award IIS and partially funded by the National Endowment for the Humanities under grant HK-50011