Design and Evaluation of an Adaptive Incentive Mechanism for Sustained Educational Communities Ran Cheng and Julita Vassileva User Modeling & User-Adapted Interaction Presented by Rosta Farzan PAWS Group Meeting April 13, 2007

Problem Insufficient user participation in online communities Provide incentives to encourage participation Too much participation especially of low-quality

Goal Proposing an incentive mechanism regulating quantity and quality of user contribution and ensuring a sustainable user participation

Outline Related work Brief introduction to Comtella Proposed mechanism Collaborative quality evaluation mechanism Brief introduction to Comtella Proposed mechanism Implementation of proposed mechanism in Comtella Evaluation Discussion

Collaborative Quality Evaluation Mechanism Active and sustained user participation while avinding information overload requires quality control Decentralized moderation Real world example Measuring the quality of journals or papers by counting the time they were cited Online communities Counting the number of clicks on each item Problem Not always a click means a positive attitude Explicit user rating E.g. rating process in Slashdot Problem – “rich get richer” Unfair score for items with insufficient attention Lower initial rating Contributed late in discussion

Requirement of Incentives Time based community need New contributions in the early period of discussion Ratings of the contribution when many contributions are collected Different users have different contribution patterns Encourage higher participation for users who contributed few high-quality resources Inhibit contributions from users who contributed many low-quality resources Overall community need

Comtella Developed at University of Saskatchewan Online community for sharing URLs of class-related web-resources (bookmarks)

Proposed Incentive Mechanism Mechanism encouraging users to rate resources Adaptive reward mechanism

Encouraging Users to Rate All users can rate others’ contribution Each user receives a limited number of rating points to give out Users with higher level membership receives more points Initial rating of zero independent of level of resource provider Ensuring all contributions have equal chance to be read and rated Summative rating Incentive: Virtual currency (c-points) Limited initial c-points Awarding users for rating resources Depending on user’s reputation Can be invested to promote the initial visibility

Sorted by c-points Summative Ratings Report duplicate, broken, or special permission links

Adaptive Reward Mechanism Hierarchical membership Adapt rewards for different forms of participation Quality of users’ participation so far Community need Personalized motivational message Stating specific performance goals Calculating adaptive rewards Community Model Individual Model

Community Model Expected # of total contributions Set by community admin Community Reward Factor Usefulness of new resources

Individual Model Contribution reputation Current membership level Sharing Average summative rating of all shared resources Rating Quality ~ Difference between the specific rating and the average of all ratings the resource gets eventually Smaller difference  Higher quality Current membership level

Individual Model Expected # of resource contribution Reward Factor

Adaptive Reward Mechanism

Implementation of proposed mechanism in Comtella

Evaluation Questions to be answered Will the users in the test group rate articles more actively? How well will the summative ratings reflect the real quality of the articles? Will the users tend to share resources earlier in the week? Will the actual number of contributions be close to the desired one? Will the users share the number of articles that is expected from them? Will the users contribute a higher percentage of high-quality articles? Will there be information overload?

Participant Class on Ethics and Information Technology Share web-articles related to the topic of each week 31 4th year undergraduate students Test group: 15 Control group: 16 Randomly assigned while controlling gender and nationality

Result Users in the test group were more active in rating articles The articles with higher ratings were more likely to be chosen by users to summarize The users in the test group were more satisfied with the summative ratings received by their articles The users in the test group tended to share resources earlier in the week No big difference between total number of shared articles across the two groups In both groups, the users’ attitude towards the quality of the articles were generally neutral No information overload problem – the overall contribution did not exceed the community need The users in the test group were more active in terms of logging on the system and reading articles – sustainability

Discussion/Limitations Choosing the Parameters for the mechanism Expected sum of contributions for each topic Threshold for reputation values Community reward function Narrow scale of rating (+1 or -1) Less cognitive load No option to rate different aspect of a paper Popularity of topic, originality, interestingness Measure of quality Average rating means average taste rules In educational context tends to be superficial, easy to read articles

Discussion/Connection to Our Work Trying personalized motivational message in CourseAgent or CoPE Good evaluation questions “Some people are easy to be motivated by glory and recognition” Is there any cognitive tool to measure this? Compare Comtella with CoPE CoPE has the option to write summary and read summary written by others More personal benefit