Presenter: Libin Zheng, Yongqi Zhang Department of Computer Science and Engineering HKUST Date: 24/11/2015 Crowd-aided course selection on MOOC

1 New York Times: “2012 is the Year of MOOCs”; platfroms come forth continuously : Coursera, edX, Udacity……… 1. Motivation MOOC: Massive Open Online Course Function: provide online courses which are openly accessible via the web. Online Students who complete the courses can finally earn a certificate with some payments. Problem: low completion rate of courses, high drop rate of students. Top ten reasons for dropping out: poor course design, hidden cost, over-worklaod, lecture fatigue…….

2 1. Motivation Lack of support for students’ rating and comments on courses Students have limited access to the quality of courses before enrollment. Crowd: experienced students Course reputation user Course query Courses Course selection Enroll Highly-rated Courses Crowd-aided course selection: With the help of experienced students, users don’t have to enroll multiple courses to make the comparison by themselves during enrollment.

3 2. Problem formulation I am interested in Machine Learning, and want to register a ML course on Coursera. Which course should I select ??? Which course do you recommend? A ： B ： Comparison task post

5 2. Problem formulation Task assignment constraints Constraint: for a comparison task (A,B), only students who have experience in enrollment of both A and B can do the comparison.  Some comparison tasks(A, B) cannot be assigned for there is no valid responders.  Even the task(A,B) is assigned, we may not receive its answer due to: the student simply do not respond to a vote. it takes an unacceptably long time to respond. Only partial evidence is offered ！！ Result of a vote matrix W(i,j) is the number of votes for o j being greater than o i Judgment problem: With vote matrix W, what is the best estimate for the max ?

3. Methods Maximum Likelihood(ML) ML: using Bayesian formulas to determine the max element, which requires to enumerate all the permutations(NP-Hard! Not Practical). Object j is the true max element

3. Methods Local strategy(local) Define a score for each object, initialized as: score(i) = wins(i) – losses(i) Differentiating the votes by considering the strength of the object o i was compared against. That is, o i would be given bonus if it wins a strong candidate.

3. Methods Iterative strategy(ITR) Iteratively prune half of the objects with lower scores, and update the scores with only the remaining. Considered. Define a score for each object, defined as: dif(i) = wins(i) – losses(i) The final survivor is considered as the max.

3. Methods pageRank The probability of object i’s being the max element is the sum of probabilities of the defeated objects A modification of the classical PageRank algorithm For each object, firstly calculate its period, and then compute its average probability over that period, serving as the final score.

4.1 Synthetic Experiment Experiment setup Control variablesValues From 1 coverage to 5 coverage Worker accuracyFrom 0.55 to 0.95 Number of runs5000 Construction of W Average as the result

Evaluation criteriaFormulation PrecisionRatio of the true max elements among all returns. MRR (Mean Reciprocal Rank) The inverse rank of the true maximum in the predicted ranking. Higher value indicates better performance 4.1 Synthetic Experiment Experiment setup Control variablesValues From 1 coverage to 5 coverage Worker accuracyFrom 0.55 to 0.95 Number of runs5000 Construction of W Average as the result max descending Reciprocal Rank = 5/7 True order

4.1 Synthetic Experiment Precision versus coverageMRR versus coverage Compare ‘ML’ with other method; since running ‘ML’ is very costly, we only consider an input of 5 objects with worker accuracy = 0.75.

4.1 Synthetic Experiment Number of objects: 100Coverage: from 1 to 10 P ： 0.75 P ： 0.55 P ： 0.95

4.2 An off-line experiment The questionaire ： From 16 volunteers, we obtain a vote matrix consisting of 46 votes, which is more than 2 coverage. Result:

4.2 An off-line experiment ITR: rank = Local: rank = The reasonable way to define the ground truth.

Thanks ! 10